Sr Data Platform Software Engineer POST NUMBER: 459091
Job Description:
We are seeking a skilled Senior Data Platform Software Engineer to join our Data Platform team in Kansas City, MO. In this role, you will be instrumental in designing, coding, implementing, and optimizing a cloud-native data stack that leverages best-in-class open-source tools. The ideal candidate will design, build, and maintain an opinionated, resilient, and scalable data platform in a private cloud environment—enabling data-driven decision-making, analytics, and machine learning, while providing deep insights out of the box. This role blends data engineering, software development, and infrastructure management, leveraging tools such as Apache Iceberg, Java, Airflow, Spark, Kafka, and Superset.
Core Responsibilities
Data Pipelines
- Develop and maintain robust, fault-tolerant data ingestion and transformation pipelines using Java, Python and Spark.
- Define flexible and scalable data schemas using Apache Iceberg.
- Support both batch and real-time data processing, including integration with Apache Kafka.
- Ensure reliability, observability, and integrity of data pipelines.
- Design and implement scalable, secure RESTful and data APIs for data access and integration.
- Build APIs for data ingestion, transformation, and consumption across internal services.
- Ensure API performance, consistency, and proper access controls.
- Apply best practices for API design, versioning, and Swagger documentation.
- Integrate APIs with orchestration tools like Apache Airflow and metadata platforms.
- Enable seamless interoperability between data consumers, pipelines, and governance systems.
- Evaluate and implement metadata management platforms such as DataHub, Apache Atlas, or OpenMetadata to support data cataloging, lineage, and governance use cases.
- Collaborate with data stakeholders to align metadata solutions with organizational needs.
- Define and enforce governance policies related to data quality, privacy, and compliance (e.g., GDPR, CCPA).
- Implement fine-grained access controls, encryption, and auditing with a focus on regulatory compliance and data traceability.
- Automate data workflows, infrastructure provisioning, and deployments using tools like Airflow, Ansible, Salt and Kubernetes.
- Implement CI/CD pipelines for data platform updates and enhancements.
- Optimize data storage and queries using Apache Iceberg and Spark to ensure high performance and low-latency access.
- Identify and address performance bottlenecks; implement partitioning, caching, and indexing strategies.
- Monitor data platform health using tools such as Prometheus and Grafana dashboards.
- Configure real-time alerts to proactively detect and resolve pipeline failures or data issues.
- Troubleshoot and resolve platform outages and data incidents promptly.
- Work closely with data scientists, analysts, and engineers to understand data needs and deliver performant, scalable solutions.
- Collaborate with cross-functional teams (Cloud Engineering, Network, and DevOps/Solutions Engineering) to troubleshoot and resolve infrastructure issues.
Education
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field.
- 5–8 years of experience in data engineering, with a strong focus on cloud-based data platforms.
- Strong programming skills in Java.
- Deep knowledge of Apache Iceberg, Spark, Superset, and Kafka.
- Familiarity with metadata management platforms like DataHub, Apache Atlas, or OpenMetadata and experience with their evaluation or implementation.
- Experience with cloud-native infrastructure tools such as Kubernetes, Ansible, Salt, etc.
- Strong analytical and problem-solving skills.
- Effective communication and collaboration with cross-functional teams.
Determining compensation for this role (and others) at Vaco/Highspring depends upon a wide array of factors including but not limited to the individual’s skill sets, experience and training, licensure and certifications, office location and other geographic considerations, as well as other business and organizational needs. With that said, as required by local law in geographies that require salary range disclosure, Vaco/Highspring notes the salary range for the role is noted in this job posting. The individual may also be eligible for discretionary bonuses, and can participate in medical, dental, and vision benefits as well as the company’s 401(k) retirement plan.
Vaco by Highspring values a diverse workplace and strongly encourages women, people of color, LGBTQ+ individuals, people with disabilities, members of ethnic minorities, foreign-born residents, and veterans to apply.
EEO Notice
Vaco by Highspring is an Equal Opportunity Employer and does not discriminate against any employee or applicant for employment because of race (including but not limited to traits historically associated with race such as hair texture and hair style), color, sex (includes pregnancy or related conditions), religion or creed, national origin, citizenship, age, disability, status as a veteran, union membership, ethnicity, gender, gender identity, gender expression, sexual orientation, marital status, political affiliation, or any other protected characteristics as required by federal, state or local law.
Vaco by Highspring and its parents, affiliates, and subsidiaries are committed to the full inclusion of all qualified individuals. As part of this commitment, Vaco by Highspring and its parents, affiliates, and subsidiaries will ensure that persons with disabilities are provided reasonable accommodations. If reasonable accommodation is needed to participate in the job application or interview process, to perform essential job functions, and/or to receive other benefits and privileges of employment, please contact HR@vaco.com .
Vaco by Highspring also wants all applicants to know their rights that workplace discrimination is illegal.
By submitting to this position, you agree that you will be giving Vaco by Highspring the exclusive right to present your as a candidate for the foregoing employment opportunity. You further agree that you have represented information about yourself accurately and have not affirmatively misrepresented your qualifications. You also agree to maintain as confidential, to the fullest extent permitted by law, any information you learn from Vaco by Highspring about the position and you will limit disclosure of information about the position only to the extent necessary to perform any obligations in furtherance of your application. In exchange, Vaco by Highspring agrees to exercise reasonable efforts to represent you through all solicitation, job screening and resume dispersal.
Privacy Notice
Vaco by Highspring and its parents, affiliates, and subsidiaries (“we,” “our,” or “Vaco by Highspring”) respects your privacy and are committed to providing transparent notice of our policies.
- California residents may access Vaco by Highspring HR Notice at Collection for California Applicants and Employees here.
- Virginia residents may access our state specific policies here.
- Residents of all other states may access our policies here.
- Canadian residents may access our policies in English here and in French here.
- Residents of countries governed by GDPR may access our policies here.
Pay Transparency Notice
Determining compensation for this role (and others) at Vaco by Highspring depends upon a wide array of factors including but not limited to:
- the individual’s skill sets, experience and training;
- licensure and certification requirements;
- office location and other geographic considerations;
- other business and organizational needs.
With that said, as required by local law, Vaco by Highspring believes that the following salary range referenced above reasonably estimates the base compensation for an individual hired into this position in geographies that require salary range disclosure. The individual may also be eligible for discretionary bonuses.