Data Pipeline Specialist, Senior

Key Role

Work with a team using an Agile methodology to provide high-quality service to our customers through the design and development of enterprise data flows using Apache NiFi to migrate data from identified source systems to an enterprise data lake managed using Hortonworks HDP. Confer with customers and data stewards to identify, define and implement the functional and technical rules needed to manage the data within the lake, to define the requirements and algorithms needed to transform transactional data into analytical data objects, and to develop pipelines to transform source data into analytical data objects that can be used by customers and analysts to visualize trends and promote analysis of the data. Interact with other data pipeline engineers and data stewards to build and manage workflows used to ingest, profile, cleanse, and curate data and ensure workflows are built according to defined standards and consistent with pipeline specifications and customer data requirements. Employ an understanding of basic data flow concepts, flow-based programming, and flow-based models and apply knowledge of existing programming languages like java, python, and Apache Spark. Collect metadata required to maintain lineage and provenance of the data from the time it leaves the source, and through the various levels of curation performed within the data lake. Document data management designs, solutions, processes, and procedures and present them to teammates when needed. Apply knowledge of data security, ethics surrounding the handling of sensitive data, exception handling, data governance, metadata management, change management.

Basic Qualifications:

  • 2+ years of experience with Extract Transform Load (ETL), data migration, data quality, metadata management, and data integration in Cloud environment or AF data warehousing environment
  • Experience with development using python, java, or scala for use with Apache Spark
  • Ability to gather and analyze internal application data needs by means of interviews, workflow analyses and facilitated discussions with users
  • Ability to translate data needs into detailed functional and technical designs for development, testing and implementation
  • Ability to create complex data loading routines to create operational data stores, data warehouses, or data lakes to satisfy functional or technical requirements 
  • Ability to configure NiFi components in line with design requirements as well as peer review
  • Ability to identify and communicate risks and issues affecting business rules, functional requirements and specifications
  • Ability to ingest data into HDFS or Hive using JDBC and Sqoop data connectors
  • Active Secret clearance
  • HS diploma or GED and 5+ years of experience with Extract Transform Load (ETL), data migration, data quality, metadata management, and data integration, or BA or BS degree

Additional Qualifications:

  • Experience with Cloud-based or open-source data pipeline technologies including Data lakes, HDFS, Hadoop ecosystem of products (Hive, HBase, Spark, Sqoop), NiFi, Apache Spark
  • Experience with programming languages such as python, java, or scala
  • Knowledge of USAF Logistics domain business needs and data environments
  • Ability to develop and incorporate exception handling flows that will handle data management that does not conform to provided specifications
  • Ability to design and develop data profiling, data quality, and discover routines needed to identify sensitive information within the data being handled within the environment
  • Ability to serve as a liaison between technical, quality assurance, and non-technical stakeholders throughout the development and deployment process
  • Ability to work with various levels of project managers, modelers, data stewards, and architects to design data loading processes and identify potential problem areas
  • Ability to establish and maintain a high level of customer trust and confidence
  • Ability to lead through influence and deliver results through others
  • Ability to bring creative approaches to problem-solving and focus on details while simultaneously maintaining the "big picture" view
  • Possession of excellent interpersonal skills, including mentoring, coaching, collaborating, and team building
  • Possession of excellent decision-making ability, balancing what is right with what is realistic
  • Possession of strong verbal and written communication skills for a wide variety of audiences including proven ability to deliver conference presentations


Applicants selected will be subject to a security investigation and may need to meet eligibility requirements for access to classified information; Secret clearance is required.

We’re an EOE that empowers our people—no matter their race, color, religion, sex, gender identity, sexual orientation, national origin, disability, veteran status, or other protected characteristic—to fearlessly drive change.


Not ready to apply? Join our talent community and sign up for job alerts.