Job scope
At Data for Public Good, we are building an advanced state of the art HPC analytics processing engine. The role involves developing a full stack analytic system including workflow/process graph compilations, workflow orchestration, distributed process execution, distributed file storage and system level performance optimization with TF/Torch/Cuda and other computing acceleration libraries, and developing an api server and visualisation utilities to interact with the HPC backend. The Analytic Engine will be used in a highly scaled national level Geospatial Engine storing and processing large data such as satellite imagery.
Good to see on resume
- First hand experience in MLOps on providers such as databricks, AWS (Sagemaker), Azure, Kubeflow, etc.
- First hand experience with writing optimized Python code using libraries like asyncio, dask, etc.
- Experience with technicalities of data visualization tools such a superset (internals, code modification, etc) and API servers for data science based applications.
- First hand experience building self hosted analytic clusters with Dask or Spark.
- Good experience in optimizing ML workflows on Pytorch / TF with system level improvements such as prefetching, shared data, Gpu Direct Memory, etc would bump up your visibility.
- Experience developing software in Cloud Native approach will also bump up your visibility, for e.g exposure to Kubernetes (CRDs, Operators), Istio, Argo, NATS, etc