Guide or docs for setting resource spec for KubeflowV2DagRunner?

[jeongukjae]

Hi, there.

Is there any guide or best practice to set custom resource specs (CPUs, RAM, GCE node spec, …) for KubeflowV2DagRunner? I checked the related issue and protobuf specs.

• Issue: Request cpu / memory for containers in Kubeflow · Issue #3194 · tensorflow/tfx · GitHub
• Proto spec that KubeflowV2DagRunner.run returns: pipelines/pipeline_spec.proto at master · kubeflow/pipelines · GitHub

It works, but I think that it is not a clean or good way to set the specs compared to the V1 Runner way, since it relies on undocumented JSON data. (In V1 runner, It supports pipeline_operator_funcs arg in KubeflowDagRunnerConfig.)

Thanks

[chunduriv]

@jeongukjae,

You can pass the Custom CPU and Memory in with_platform_config as shown in b15d592

For Example :

    bq_example_gen = big_query_example_gen_component.BigQueryExampleGen(
        query=query).with_platform_config(
            pipeline_pb2.PipelineDeploymentConfig.PipelineContainerSpec
            .ResourceSpec(cpu_limit=5.0, memory_limit=10.0))
    deployment_config = pipeline_pb2.PipelineDeploymentConfig()

Thank you!