Senior Systems Engineer #339

Summary The Senate Sergeant at Arms is seeking a Senior Systems Engineer. The complete vacancy announcement and application can be found on the United States Senate Career Page at https://sen.gov/1VNZQ. This vacancy announcement closes at 7pm EST on the closing date. Late applications will not be accepted. Learn more about this agency Duties Contributes to the section's work effort in responding to incidents from other sections, including the help desk. Supports tasks which may include helping to develop standard operating procedures, assignments, and coordination of work efforts to address issues. Assists users with computer needs such as installing and configuring software/hardware on their PC; troubleshoots problems with user systems. Provisions new accounts for users. Supports the section's work effort to develop processes and procedures to streamline the resolution of issues. Provides technical support for Senate information technology or telecommunications services, systems, hardware, software, and storage solutions to the users; supports Senate offices, contract support vendors, and other technical staff in troubleshooting and managing the various Senate environments. Maintains and monitors LAN/WAN and SAN operations and solves LAN/WAN and SAN applications problems; installs integrates, customizes, enhances, troubleshoots, and tests LAN and SAN hardware and software. Assists in installing, upgrading, customizing, tuning, and maintaining databases for optimal performance and availability; assists in developing, testing, and implementing database store procedures, functions, triggers, views, and packages. Conducts acceptance and integration testing; assists vendor systems implementation by troubleshooting related system issues affecting Senate users. Analyzes, designs, develops, implements, and maintains applications to meet the needs of Senate users; assists with product evaluations and implementation. Attends internal training provided by subject matter experts to understand operational tasks, documents support needs and share the information within the assigned section. Other duties as assigned. Requirements Conditions of employment For conditions of employment and citizenship requirements, please visit the job announcement on the United States Senate Career Page at https://sen.gov/YO0NL. Qualifications Other Duties/ ResponsibilitiesParticipates in on-site and on-call rotation.Required Work ExperienceFive to eight years of relevant work experience (GED), three to six years (Associates), one to three years (Bachelor’s), one to two years (Master’s).Required Special Skills/ KnowledgeAs part of our hiring process, we may conduct a skills assessment to better understand an applicant’s proficiency in key areas relevant to the role. Manages current storage capacity and ensures storage is available to support all computing environments; plans for future storage growth; monitors and confirms that system files and application products are successfully backed up; performs necessary installations, patches, and upgrades. Works escalated issues from lower-tier team members. Mentors/relays information on solutions to train others. Analyzes new hardware and software releases for their impact on the Senate environment; tests new releases on Senate supported platforms; develops product configuration standards for use in the Senate environment.Creates, develops, modifies, instructs and provides technical assistance on business continuity plans and processes.Desired Qualifications Extensive experience serving in a Splunk administration or engineering role withina complex enterprise environment.Demonstrated experience building, implementing, and administering distributed Splunk environments on Red Hat Enterprise Linux (RHEL) operating systems, including versions 9, and 10.Hands-on experience supporting clustered Splunk architectures, including heavy forwarders, indexers, and search head clusters, with responsibilities spanning security, performance optimization, and operational support.Proven ability to edit, maintain, and optimize Splunk configuration files and applications in production environments.Demonstrated experience onboarding and normalizing data into Splunk using forwarders, deployment servers, scripted inputs, TCP/UDP inputs, and modular inputs from diverse data sources.Experience developing and integrating application interfaces using REST APIs.Proficiency in scripting and automation using technologies such as Ansible,Python, and JavaScript, including the use of automation frameworks to manage infrastructure and deployments.Demonstrated experience operating across multiple technology stacks, applying sound engineering principles beyond a single platform, and integrating Splunk with complementary enterprise systems.Experience evaluating, implementing, or supporting emerging technologies in support of operational, security, or data analytics initiatives.Proven ability to identify opportunities for technical innovation and translate them into practical, scalable solutions aligned with organizational objectives.Working ConditionsThe U.S. Senate network cannot be taken offline for maintenance during the workday, or while the Senate is in session. As such, maintenance windows may only occur at night, on weekends, and occasionally on holidays. Employees who perform systems upgrades, maintenance, wiring, backups, support our alternate data centers will have schedules that include working nights, weekends, and holidays. This position directly supports essential services of the U.S. Senate. As such, this position requires the employee be available and prepared to work during a lapse, in inclement weather, on holidays, weekends, and during late nights to ensure essential services to the Senate continue without interruption. In the context of government furloughs, this position is considered excepted.Since this position requires onsite presence in support of the U.S. Senate when in session, this position requires the employee to be available and prepared to work during a lapse, in inclement weather, on holidays, and during late night, overnight, and weekend sessions. In the context of government furloughs, this position is considered excepted. Security ClearanceThis position requires that the applicant obtain and maintain a Secret U.S. Government security clearance.Applicants must be U.S. citizens in order for the SAA to submit your application for a security clearance. Education High School Diploma/ GED. Additional information The Sergeant at Arms is an equal opportunity employer in accordance with the requirements of Senate rules, regulations, and applicable Federal Laws. This agency provides reasonable accommodations to applicants with disabilities. If you need a reasonable accommodation for any part of the application and hiring process, please notify the agency. Decisions to grant reasonable accommodations will be determined on a case-by-case basis. Please email [email protected] with “Applicant Accommodation” in the subject line. No moving, relocation or pre-employment travel expenses will be paid for this position, or while in application for this position. Expand Hide  additional information Candidates should be committed to improving the efficiency of the Federal government, passionate about the ideals of our American republic, and committed to upholding the rule of law and the United States Constitution. Benefits A career with the U.S. government provides employees with a comprehensive benefits package. As a federal employee, you and your family will have access to a range of benefits that are designed to make your federal career very rewarding. Opens in a new windowLearn more about federal benefits. Eligibility for benefits depends on the type of position you hold and whether your position is full-time, part-time or intermittent. Contact the hiring agency for more information on the specific benefits offered.