Outsourcing Virginia’s IT operations to Northrop Grumman was supposed to put the state’s network beyond politics, overhaul and standardize it and improve operations, but the effort has failed in almost every respect.

The $2 billion, 10-year contract with Northrop Grumman calls for a network overhaul – or transformation as the state calls it –is months behind scheduled despite repeated deadline extensions.

Network services to agencies have been cut off or have been poor enough to affect the services agencies deliver to the public.

Members of boards are overstepping their authority to negotiate with Northrop Grumman and the chain of command is so fragmented that it’s difficult to get any one entity in control of the problems.

This is all according to a report issued this week by the states Joint Legislative Audit and Review Commission (JLARC), which points a finger at Northrop Grumman, but also at boards, agencies and individuals for problems like these:

• In May, Virginia State Police in Newport News lost Internet access for 78 hours, affecting the ability to perform its duties.

• Also in May Environmental Quality in Roanoke lost its network connection for 31 hours.

• In June the Department of Motor Vehicles in Bland lost the network for 31 hours, affecting customer service.

• One state correctional facility lost incoming phone service at 4 a.m. and was assigned a priority level that gave Northrop Grumman 18 hours to fix it. The priority was assigned based on the number of employees at the facility – 30 to 40 – not on the need dictated by the fact that the facility houses 1,000 inmates.

The state has tried extending deadlines, withholding payments and levying penalties in order to resolve the problems, but they persist, according to the JLARC report. As things stand now, the state could get out of the contract, which runs through 2015, but it could cost as much as $399 million to do so, and the state would still have to pay for someone else to run the network or develop state resources to do so.

The report cites benefits of the contract with Northrop Grumman. These include replacement of networking equipment – 45,200 of 57,500 PCs – to standardize the infrastructure across the state that was scheduled to be completed in last July. Replacing aging PCs actually enabled applications that older PCs couldn’t support, one large agency says.

Internet traffic is funneled through a gateway rather than hundreds of individual agency connections. A program to monitor for outages and performance and to alert the help desk to the most severe problems is underway.

A transition to a single e-mail infrastructure is underway, with 26,200 of 63,500 accounts having been migrated. The new e-mail system rides on the state network rather than the Internet and is screened for viruses and spam. The help desk has been centralized, and more online user self-help sites are available.

Even so, state agencies pan many of the services. The report says 44% of agencies find the disaster recovery and backup services are inadequate. Network services were ranked poor by 41% of agencies.

The help desk was rated poor by 48% of agencies with complaints including calls being routed to technicians with the wrong expertise and with complaints being assigned lower priority to trigger longer resolution times.

In a letter responding to the interim report, Northrop Grumman’s vice president and general manager of the firm’s Civil Systems Division Tom Shelman says the project is the first of its kind and that no other project on this scale exists. “Virginia is breaking new ground and should be proud of that fact in spite of the challenges we all acknowledge exist. Other states will follow the lead and draw on the Virginia experience.”

The work to be done includes 59 projects over 72 agencies at more than 2,000 sites and encompasses replacing PCs, servers, mainframes, e-mail, network, security, help desk and telecom. “Projects are interdependent and delays with one project can have cascading effects on other projects,” the report says.

The work was to be done by July of this year, and a corrective action plan was filed in August promising to finish by June 2010, but it leaves out four agencies. As of September, 32 of 59 projects were completed. The report says the delays happened because “agency needs have not been fully addressed or fully understood.”

“[The] largest single reason for delay appears to be inadequate planning by [Northrop Grumman],” the report says, and it didn’t perform adequate due diligence. Virginia’s VITA [Virginia Information Technologies Agency] also bears blame for delay because it didn’t fully understand agency needs and that Northrop Grumman wasn’t meeting them.

VITA was formed in 2002 to centralize state IT, improve services and reduce cost by $100 million per year. The following year the state created the Information Technology Investment Board (ITIB) to oversee VITA. ITIB hired a CIO who administers VITA.
VITA signs contracts and oversees projects, but it is overseen by ITIB, which falls under the governor on the organizational chart, but the governor lacks the statutory authority to jump into the contract dispute over the massive IT project, the report says.

The JLARC report says the network and services provided under the contract don’t meet the needs of some agencies, and there is no catalog of services the network offers to agencies or their price. VITA is supposed to define the network requirements and Northrop Grumman is supposed to recommend services that comply. The two parties disagree on which should identify unmet agency needs.

The report states that the responsibility is shared by VITA and Northrop Grumman, but the report finds that Northrop Grumman is responsible for the delays because its due diligence was inadequate.

The complexity of the state’s IT environment was not understood by Northrop Grumman at the time it negotiated the contract. Hundreds of sites and thousands of network assets were left out of an inventory taken in 2005 before the contract was awarded. VITA and Northrop Grumman disagree on whose fault the inadequate due diligence was.

The report found that Northrop Grumman failed to adequately plan for transformation of the network or account for the business continuity of state agencies involved. Northrop Grumman says it was barred from directly contacting agencies and that it could not get an accurate list of sites. VITA says it had appropriate access and should have known all the sites because it was providing service to them.

The transformation schedule didn’t account for making change without disrupting agency business, including failure to post blackout dates on the schedule. The report says that senior Northrop Grumman staff said it was unaware of how complex the IT infrastructure was within agencies.

An HP review of the transformation plan one year into it rated it fair to poor with an almost incomprehensible schedule. It ranked teamwork and communication as poor very and poor, and risk management was rated poor as well.

Four reviews by IT consultancy CACI between 2006 and 2008 found the schedule was driven by deadlines in the contract more than by the scope of the actual work, and that plans lacked detail and were not maturing.

Northrop Grumman says it made organizational changes in 2007 and 2008. The legislative report says agencies agree that in the fall of 2008 they got more contact with Northrop Grumman and that planning improved because of it.

In his letter to the JLARC, Shelman says Northrop Grumman has taken steps that are already improving the situation, including better communication with VITA and individual state agencies.

JLARC’s full report is due in December.