TO: AC Transit Board of Directors
FROM: Michael A. Hursh, General Manager
SUBJECT: Department of Innovation and Technology Critical Projects Update
BRIEFING ITEM
RECOMMENDED ACTION(S):
Title
Consider receiving an update on critical Department of Innovation and Technology projects.
Body
STRATEGIC IMPORTANCE:
Goal - Convenient and Reliable Service
Initiative - Infrastructure Modernization
The Department of Innovation and Technology (IT) is focused on accelerating Digital Transformation across all the District’s business operations. It has made Digital Transformation a lynchpin in the Department’s overall Infrastructure Modernization Initiative. Many IT projects and initiatives, as part of the Digital Transformation journey, are currently underway in various phases of development, deployment, testing, and completion. However, due to the COVID19 Pandemic, we are increasing our focus on response to the Shelter-in-Place Order requiring non-essential staff to Work from Home (WFH). Since the onslaught of coronavirus and considering the Governor’s State of Emergency declaration regarding the COVID-19 outbreak, the Innovation and Technology Department has stepped up emergency response and been instrumental in playing a critical role in keeping all IT systems operational. The IT team increased support capabilities to accommodate sudden and rapid growth in the number of remote workers, including emergency procurement of mobile computer hardware, remote access technology, and cloud-based anti-theft software.
A collaborative IT effort was initiated to optimize and utilize current tools to remotely monitor Help Desk phone calls and voicemails, providing seamless support regardless of the location of the technician. A fast-tracked deployment process was implemented to deploy laptops as fast as possible. New computer configurations and models were incorporated into the District’s standards to accommodate available hardware and accessories. Also, additional cell phones have been rolled out to help, essential remote users.
Primarily, IT has been instrumental in working on remote access improvements, innovating end-user collaboration tools, increasing Internet bandwidth at GO, deploying redundancy in connectivity, bolstering the cybersecurity posture, keeping up the mission-essential business applications, and improving emergency communications.
During the Shelter-in-Place directive, the IT team worked with the Scheduling and Transportation Departments in successfully implementing two back-to-back signups; one, effective March 29th, and the second effective two days later, on March 31st.
BUDGETARY/FISCAL IMPACT:
The is no budgetary or fiscal impact associated with this report.
BACKGROUND/RATIONALE:
Modernizing and upgrading various Information Technology systems at AC Transit is a key strategic initiative. Many IT projects currently underway are in various phases of development, deployment, testing, and completion. This staff report provides a brief status update on how COVID 19 has affected the following critical Information Technology projects at the District:
A. COVID-19 Work from Home Response
B. Cybersecurity Risks and Initiatives
C. Enterprise Applications and Data Management
D. Infrastructure and Resiliency
E. Website, Mobile Ticketing, and Mobile Apps
F. BRT Technology Systems
G. Computer-Aided Dispatch/Automated Vehicle Location System (CAD/AVL)
H. HASTUS DAILY
The COVID-19 Pandemic and resulting Shelter-In-Place caused significant disruption and distraction to the ongoing IT projects: however, IT Staff, along with project stakeholders, are pushing major IT projects forward. On the BRT project, IT staff is leading the Technology Systems Integration efforts in project management, system design, installation, validation, testing, and acceptance.
A. COVID-19 Work from Home Response
The COVID-19 Pandemic abruptly transitioned the District into a remote workforce. Staff suddenly did not have the necessary technical tools they were accustomed to day-to-day. The IT Support team consisting of the Help Desk and Desktop Support Team procured and deployed 100 new laptops in 6 weeks to support offsite staff. An additional 100 more are on the way to provide the necessary equipment needed to maintain day to day operations.
With our diverse workforce dispersed throughout the Bay Area, the ability to interact and collaborate while Shelter-In-Place is in effect is crucial. The IT Team quadrupled licensing for the District’s online collaboration tools to accommodate conferencing needs. Since the start of this Covid-19 crisis, online collaboration using the tools provided increased 93% from the end of February to April 30th.
With staff experiencing Work from Home for the first time, new and different technologies were introduced. In a true team effort, the various IT Teams came together to develop documentation on different technologies to educate each other and our remote staff regarding Virtual Private Networks, Cloud, Resource Access, and Cybersecurity.
To securely and efficiently support all the new technologies implemented and planned, a new remote support platform was procured to manage remote end-user devices that are outside the District’s local network. This technology will allow the IT support staff to remotely manage, maintain, and, if needed, connect to computers anywhere as long as an internet connection is present.
The District is highly reliant on manual paper-based processes. With the onset of the Covid-19 Pandemic and remote staff, the process is difficult to optimize using present methods. The IT Department, in collaboration with General Counsel and District Secretary, is procuring and adopting the use of an Enterprise-wide digital signature solution to streamline and automate workflows and standardize the approval process using well known, secure, and trusted technology.
IT, in collaboration with critical stakeholders, procured and is in the process of implementing a new cloud-based software platform for Emergency Response Management. This new software platform will automate various processes defined by the National Incident Management System (NIMS) governed by FEMA. This cloud solution will modernize the District’s Emergency Operations Center (EOC) while seamlessly integrating with Alameda County’s Virtual EOC to provide real-time updates using standardize forms based on FEMA specifications. The software platform will be a tremendous help in generating FEMA reports as well as reimbursement requests.
B. Cybersecurity Risks and Initiatives
During the COVID-19 emergency, in addition to the common threats affecting Cybersecurity workloads, including malware, brute force, system-level attacks, and privilege escalations, the phishing campaigns and scams are skyrocketing keeping the Cybersecurity team on high alert. The Cybersecurity team has been instrumental in advancing the development and adoption of cybersecurity policies and controls across the District.
AC Transit has been hit with hundreds of cyber-attacks in April and May, which include account compromises, credentialed phishing attacks, email scams, and extortion emails.
The lack of resources and cybersecurity expertise has led to weak incident responses and prevented AC Transit from having 24x7 visibility into security risks. Due to many legacy applications, and antiquated identity management infrastructure, the District lacks granular controls to block activities based on risk, manage administrative activities in Office 365, identify data moving between cloud applications, identify shadow IT, detect insider threats, and provide real-time visibility and control into cloud applications. Asset identification and vulnerability detection programs do not exist, asset count reported by current tools and technology vary widely. The existing Public Key Infrastructure (PKI) requires an immediate overhaul. The current system is old and lacking proper management and real-time visibility to the number of digital certificates issued, revoked, and expired.
To strengthen the core cybersecurity infrastructure and thwart cybersecurity attacks, the Department of Innovation and Technology is undertaking a few high priority initiatives:
• Develop and implement a comprehensive Cybersecurity Framework that will include the continuous monitoring of IT as well as all other District technology systems
• Deploy advanced threat and vulnerability assessment technology that will automate the vulnerability identification, prioritization, and remediation processes
• Strengthen existing email security to provide cyber resilience for email, protect emails from evolving threats, keep the email up and running during downtime, and reduce the time to recovery. The solution will automatically discover and analyze every malicious email in real-time as they are delivered to employees.
• Strengthen the security of current certificate authority to provide better visibility, reporting, and tracking of digital certificates.
• Develop 24x7 monitoring capability for the District to improve security incident detection and response and defend against intrusions regardless of the type of attack at any time. Managed Detection Response (MDR) will improve security incident detection through constant monitoring and analysis of networks, servers, and databases.
• Implement a solution that will examine Cloud traffic and expose Cloud threats that may evade our existing security controls. The solution will build the District’s capability to provide visibility into Office 365 applications such as Exchange, SharePoint, and OneDrive and reveal previously unknown critical activity and insights into data usage (PII).
• Implement Multi Factor Authentication (MFA) across the District so that employees can securely access resources from any location and any device allowing them to work from anywhere and be more productive.
C. Enterprise Applications and Data Management
The Work from Home (WFH) conditions necessitated deploying a short-term and swift electronic approval process, instead of approving the invoices, documents, and contracts manually. IT Staff devised an electronic workflow and provided training documentation and end-user education to convert the manual processes into a digital approval process. Additionally, extensive departmental training was provided to educate users with various online applications such as PeopleSoft, TEAMS, ZOOM, MyACT, BusTime, Ellipse, SharePoint, OneDrive, and Website.
IT staff in close coordination with Human Resources (HR), Transportation, Finance, and Payroll departments successfully implemented the new business rules necessitated due to the COVID19 Families First Coronavirus Response Act (FFCRA or Act). New COVID19 Sick and FMLA Time and Labor business rules have been developed, tested, and deployed in the District Enterprise Resources Planning (ERP) software platform of PeopleSoft.
The Emergency Schedule has been validated and published into District Business Critical Systems such as CAD/AVL, District Website, Mobile Website, and the Enterprise Database. IT Staff worked many weekends with various stakeholders in assuring the reliable and quality data feeds to Google and 511, so third party apps display correct predictions. For the first time since its implementation, the Clever Devices team implemented two back to back schedule imports during one week. The first implementation became effective March 29th. The second implementation for the Emergency Schedule Plan became effective March 31st. Both schedules were tested and implemented over the weekend and deployed with a minimal number of problems.
A new PeopleSoft Approval Workflow Engine (AWE) was implemented to create, execute, and manage approval processes with a user-friendly interface. This new implementation, de-customized, and optimized many timesheet and Personnel Requisition approval processes.
Strategic Sourcing is a PeopleSoft module that will replace AC Transit’s legacy web-based vendor access system in procuring goods and services through online bids and solicitations from prospective and existing suppliers. The implementation of the new module was initiated to transform, optimize, and automate paperless procurement business processes and deliver a self-service vendor portal on AC Transit’s public-facing Website. This module will enable the District to manage the sourcing process, which will allow streamlined Request for Quote (RFQ) and Request for Proposals (RFP) activities. The Business Requirements Specifications, Module Setup, and Unit/Regression Testing are completed, and IT staff is working on developing testing and training materials.
Supplier Contract is another PeopleSoft module that will provide the District with a new platform to create and manage transactional procurement contracts using Microsoft Word. The system will provide a structured method to develop and maintain the contract clause library and the life cycle and approval processing for documents. This project is still in the planning phase and will start with gap analysis and business requirements gathering.
The District’s PeopleSoft applications are hosted on a Private Cloud via a contract which expires in February of 2021. The procurement of a competitively awarded multi-year follow-on contract is in process. The PeopleSoft platform connects with many mission-critical operational software systems in the District and provides vital data elements. PeopleSoft is one of the core technology platforms and needs to be up and running all the time for 24 x 7 bus operations.
The IT team is working on a project to move the District from a legacy Microsoft Access Database to the Enterprise Asset Management platform, Ellipse Warranty Recovery Module. The Warranty Recovery Implementation project will automate manual warranty recovery processes, eliminate all paper-based warranty processes and track additional warranty types not currently being tracked. Clearly defined roles and responsibilities are being created for repeatable, sustainable, and scalable warranty recovery business processes. IT team in collaboration with Finance and Maintenance Departments is working with Ellipse Hosting and Managed Service provider, AddOns, to complete the Gap Analysis Documentation on the functional requirements. The vendor is also working on the Setup and Configuration of the Ellipse Warranty module.
D. Infrastructure and Resiliency
It is noteworthy to mention that during the COVID-19 emergency, the District IT infrastructure remains fully operational, and is being monitored around the clock. The Internet bandwidth at the GO has been upgraded four times to alleviate traffic congestion and allow remote users to connect and access various District applications and data. All personal user data that was stored on "U" network drives, hosted locally at the GO, has been successfully migrated to Microsoft Office365 Cloud infrastructure. Departmental drives are being migrated to the SharePoint Cloud, one Department at a time. Once completed, all District departmental data will be in the Cloud and available online for staff to access and collaborate.
In preparing for any unforeseen emergency, IT staff worked on completing the backup Operations Control Center (OCC) setup at D2. Two Clever workstations with new console furniture have been installed at the backup OCC. These workstations are fully functional and are ready and available for use by Operations Controllers in an emergency.
IT staff is implementing a redundant Internet circuit at the GO through a new Internet provider. This new circuit, in conjunction with our existing AT&T Internet circuit, will provide continuous and reliable Internet connectivity for the GO and the Divisions. Also, redundant Internet circuits will be installed at the Divisions. Division Internet circuits will ensure survivability for data and voice communications in the event of a Wide Area Network connectivity failure.
IT staff is setting up our presence in a Private Data Center in Oakland. This Data Center will provide secure high-speed data connections from the GO and the Divisions to the District’s Azure Cloud Data Storage, to the District’s Disaster Recovery site, and to the District’s CAD/AVL system and Voice over Internet Protocol (VoIP) communications provider.
New server technology is being installed at the new Data Center which will refresh our end of life server technology and provide a secure and reliable infrastructure for production servers that cannot be virtualized in Microsoft Azure Cloud infrastructure.
The District’s new phone switch upgrade is almost complete. This project updated the Cisco Call Manager software, installed and configured new voice servers, retired our legacy Primary Rate ISDN trunks in exchange for new redundant SIP trunks, implemented new remote capable softphone software, and replaced all our legacy telephone equipment with new high-speed desk phones.
IT Staff is adding a high-speed communications path between the GO and CMF using our new BRT fiber optic cable. This new data path will use our dark fiber and provide redundancy if our primary BRT communication trunk fails.
E. Website, Mobile Ticketing, and Mobile App
In December 2018, the Board authorized the District to award a contract to Planeteria to modernize and maintain an updated Website. This Website would be mobile-centric, improve multilingual and ADA compliance and integrate new technology such as Clever Real-Time bus tracking, Granicus Legistar for Board meetings, and Peoplesoft Strategic Sourcing for vendor and bid management. There were compounding delays during the design phase of the project. Currently, the staff is working on migrating existing content. The migrated content will then be edited and formatted to the new design standards. The launch is targeted in the late summer of 2020.
The District is developing its official mobile application to assist riders with the real-time bus arrival time. This App has a unique feature of allowing passengers to submit feedback and lost and found tickets through their mobile devices. The App also allows riders to locate bus stops and information via a map. The Mobile App will be launched for Beta testing in the late summer of 2020.
The District also awarded a contract to Token Transit for mobile ticketing. Token Transit will integrate with Transit App and AC Transit’s official mobile App. All current AC Transit fare options will be available for purchase through Token Transit. Staff expects mobile ticketing to be available when AC Transit resumes collecting fares.
F. BRT Technology Systems
The Department of Innovation and Technology has provided technical expertise during all the BRT project phases from the specification and design phases to the current completion of construction and communication systems. The BRT platform-based communications equipment, fiber networking infrastructure, and software systems are supported primarily by IT staff. The current project activities include troubleshooting, witnessing tests, and in some cases, supporting the vendor installation of the public address system, Closed Circuit Television (CCTV) system, forward facing lane enforcement camera system, on-board Transit Signal Prioritization (TSP) system, real-time information signs, Clipper card readers, fiber network, video management system, system alerting and control system, and ticket vending machines.
The BRT systems have physical hardware on each platform and operational software so that Bus Operators, riders, and bus service are implemented at a reduced risk. The IT Department’s objective is to ensure that BRT communications technology is delivered as designed and that the functions and features will be successfully deployed.
G. Computer-Aided Dispatch/Automated Vehicle Location System (CAD/AVL)
IT staff in concert with the Clever Devices team, installed two complete workstations with new console furniture at the former Operations Control Center located at D2. These workstations are fully functional, able to operate the full suite of CleverCad applications as well as communicate with vehicles.
The Clever Devices team recently re-started their efforts toward system acceptance after a moratorium due to the Shelter-in-Place directive. Their first initiatives included a significant system upgrade and the installation of new hardware at the Las Vegas and Austin data centers to improve system performance.
To close this project, AC Transit expects to approve the final system acceptance of Phase I in late August 2020. Staff will recommend conditional acceptance of Phase 1 in July of 2020, followed by acceptance of Turn by Turn directions by August 31st of 2020. After Phase I system acceptance, the District expects acceptance of Phase II by late 2021. Phase II includes Secure Bus Technology and HASTUS Advanced Integration, which are both dependent on the completion of HASTUS Daily in late Summer of 2021.
H. HASTUS DAILY
Implementation of the HASTUS Daily project to replace the antiquated Operator Timekeeping System (OTS) continues to move through software testing. The OTS system is more than 25 years old; every component of the system is well beyond the manufacturer’s announced product end-of-life. As a result, neither hardware nor software components is supported by the manufacturer. The system is a major operations and security risk and needs to be replaced by the HASTUS DAILY software platform.
The COVID19 Emergency Action Plan “special signup” has temporarily delayed this final test period at Division 4. However, a HASTUS Core Group meets weekly to prepare for that final parallel testing and the HASTUS Executive Steering Committee continues driving the schedule and project implementation.
The HASTUS Daily hardware and desktop infrastructure have been upgraded so that the user systems are ready. The HASTUS Daily Crew and Daily Vehicle basic software applications have been accepted by the District. What remains to be thoroughly tested is the vehicle and employee data exchanges, and, most importantly, the weekly export of Bus Operator payroll files for consumption by PeopleSoft. Plans for a parallel testing period at Division 4 will include using HASTUS Daily and transmitting sample pay calculations while generating actual Operator pay from the Bus Operator Timekeeping System (OTS) production system.
Since the parallel test period requires starting on day 1 of an Operator signup period, identifying a start date has been challenging given the changes around the COVID-19 schedule. The current project plan is to initiate the parallel test period with the Fall 2020 signup.
HASTUS Daily Crew and Daily Vehicle modules will be accepted at the end of this Division 4 parallel testing period as well as on the acceptance of the payroll reviews conducted by the District Payroll Department. Implementation for the remaining bus divisions will follow that acceptance based on available testing and staff resources at the remaining bus divisions.
The BIDWeb project continues to be received well by Bus Operators and requires ongoing support from the IT Networking and Development teams. Improvements to the computer infrastructure during the fiscal year and internal programming has resulted in a more stable application during the Bus Operator Sign-up period. The improvements have resulted in increased use of the BIDWeb application. For example, 90% of Bus Operators used BIDWeb during the Spring 2020 signup, which is a remarkably high adoption rate for any technology.
The next step of implementing BIDWeb includes preparing this HASTUS Software module for the August 2020 general signup. The Transportation Department is working with the IT Department to finalize the scope of that effort. This improvement is envisioned to be funded with the existing GIRO annual maintenance fees.
ADVANTAGES/DISADVANTAGES:
This report is to inform the Board of ongoing critical Innovation and Technology projects at the District and to emphasize the alignment of these projects with the District’s Strategic goals.
ALTERNATIVES ANALYSIS:
There is no alternative analysis needed because this is an informational only report.
PRIOR RELEVANT BOARD ACTION/POLICIES:
None
ATTACHMENTS:
1. IT Presentation
Prepared by:
Sandra Lewis Williams, Senior Project Manager
In Collaboration with:
Manjit K. Sooch, Director of Systems and Software Development
Mike Carvalho, Enterprise Network Engineer
Lyell Amora, Computer Ops Administrator
Patricia Broadbent, Senior Project Manager
Darrell Takara, Project Manager
Tas Jalali, IT Manager, Cybersecurity
Approved/Reviewed by:
Ahsan Baig, Chief Information Officer