How Linux Saved A Fast Food Giant.

Posted by TheRealEdwin on May 17, 2010 in HowTo, Tech | 57 comments

I am a Windows guy. I have always used Windows at home, work, school, everywhere with the exception of my phone (iPhone now Nexus One) and one Linux class at FIU. I have an A+ and MCTS in Windows Vista. Soon I will have my MCITP. I drink the kool-aid. But Linux saved me and the company I sub contract to, a large fast food giant, from near-total disaster. Last month McAfee posted a virus definition update that flagged SVCHOST.EXE as a virus. This is my story of what happened.

Windows needs svchost.exe to live. It’s like removing the transmission or engine of a car. Without it, each of those workstations, at best, had no networking capabilities or, even worse, were unable to boot entirely. You need svchost.exe, otherwise you are not going anywhere. On Wednesday morning, our help desk started logging a high volume of calls in which our drive-through computer was rebooting and not loading back into Windows XP properly. Can you imagine how every IT guy felt when they realized that every single one of their XP machines just decided to reboot and not load Windows XP properly? By Wednesday afternoon, McAfee had already handed us a fix that needed to be run on the system- a SUPER DAT file, as they call it. But how do you deploy a manual fix to nearly 700 restaurants spread across the continental US and Canada? Not to mention that none of these machines have keyboards, as they are touchscreen POS systems. So we needed a tech to deploy the fix manually. To every restaurant. Do you realize how many restaurants a giant fast food company has in the world? A lot. Even with carpet in our office, I heard people shitting bricks when we realized the magnitude of the problem and the possible ways to fix it. Thankfully the franchise restaurants do not use McAfee, so they were spared.

How did this get past McAfee QA? I can imagine how it happened, but it still blows my mind that there are no fail-safes in place at McAfee to prevent this. If there are fail-safes, how many are there and how did this issue get past all of them? Either way, McAfee has a lot to answer to. So how do you fix a problem in which it would take considerably amount of money and man-hours? Only one thing came to my mind.

The touchscreen is useless while the machine is still booting up because in the BIOS, no drivers have been loaded yet. How were we going to run the fix if we can’t boot into safe mode? Our team had three ideas before I was able to help out (I was working on another assignment that was time-critical). One of the team members was testing a manual solution with a restaurant manager on the phone, and came to a happy and successful solution. We had a confirmed working solution that would take about twenty minutes of time, but it worked.

Let’s see. 20 minutes x 700 or so restaurants / 60 minutes in an hour equals to 233.3333333333333 man-hours. Holy Balls! That’s just my department too, since we only cover the US and help Canada. To think about the amount of man-hours we would have to put in to fix all of these worldwide is just incomprehensible to me.

Using some of our clout, we contacted Microsoft to see if they had anything that could help us, and they gave us a WinPE image that we could customize to run a script that would automatically fix the situation. Great, but we have two problems. First of all, we can’t run Microsoft’s proprietary .wim on our existing PXE environment. Secondly, the file size? Huge! We would have to push out nearly a CD’s worth of data to a lot of restaurants that do not have a great data connection under the best of circumstances (thanks telco duopolies!). Some of our more rural restaurants had connections so bad, they were going slower than a 56k. We were at the mercy of a piss-poor national broadband system. We pretty much ended up having to save this as a last resort, because it was basically unfeasible to send that kind of data to all affected restaurants. Some restaurants had to resort to buying a USB thumb drive and driving to another restaurant to get the file. Their connections were so bad, it could not handle a 62 MB file. Just awful.

Our last choice was to re-image the POS system using our existing Ghostcast server infrastructure. While mostly automated, each one had to be kicked off manually in each restaurant and took nearly an hour. We wanted to avoid this at all costs just because of the huge amounts of hassle involved. So how does Linux get involved? My idea was to create a small, self-extracting, PXE-bootable Linux system, mount an existing shared folder on the server in the restaurant, mount the the workstation’s Windows partition with read/write access, delete the broken svchost.exe and the virus definition, copy over a working svchost.exe, and finally reboot the machine. Logically it could work and it meets all the criteria. Small, fast, and—most importantly—fully automated.

Damn Small Linux 4.4.10

I looked at using Damn Small Linux which I had known about for a while, but sadly does not include video drivers for the POS systems we had. Should anything go wrong with the script, there was no way for us to know. After some searching via Google, I stumbled upon PLoP Linux with instructions on how to get it to work via PXE. After some hours of gnawing, scripting and fighting lack of sleep, I had a working proof of concept by 1 AM Thursday to show others that it is possible, and it clocks in at 70 MB uncompressed compared to the WinPE of 250 MB or more. I showed it to one of my superiors and he called in a few other people. I broke the POS system in our lab and showed them from start to finish how it worked. It was pretty much decided right then to use this method over the other three. After cleaning up the processes and adding several easter eggs, it was time to pilot it at 10 local restaurants. So while two team members went driving off all over Miami we wanted to compress the files as much as possible. Using WinZip, the total package actually grew to 71 MB. How does a compression program add a megabyte to the package? I have no idea. Off I was again to use my old friend 7 Zip and to create a self extracting zip of now 62 MB. Not bad. Now the real magic happens.

We have a scripting guy who is damn good. He created a script that would automatically distribute the self-extracting 7 zip file with PLoP Linux to all of our restaurants, open it, turn on Tftpd32 which then boots into PLoP running my script, and upon rebooting into XP would kill the Tftpd32 PXE server and check that the bad DAT file from McAfee was indeed gone. After that it would report a successful fix to our SQL reporting server, where we could run reports from. A lot of my co-workers were amazed at my creative approach to solving this problem, but I was just as amazed, if not more so, at our scripter’s ability to control Windows via VBScript, Powershell, and other tools.

Some final thoughts:

What does it say to you that you have to stumble around to achieve in Windows what can be easily done in Linux? What about vice versa for me with regards to how my co-worker skillfully scripted out an entire deployment and verification process when I had no idea how to even do so? I knew what needed to be done in theory, but no idea how to execute it. To me, this taught me that you should use the right tool for the right job. To seek out knowledge and try out different things. If I had not flirted with Linux during college, where would we be right now? A month later and god knows how much money, we probably would just be finishing fixing everything up. Instead, with other complications, we finished in just under a week due to issues not related to either my script or my co-worker’s VBScript. How come Microsoft’s tools don’t offer this kind of ease of use and flexibility? Or maybe they do and I have no idea they exist. Go on and experiment. It will be good for you.

I’ve since revived my old ASUS Z71v laptop with Ubuntu 10.4 Lucid Lynx for daily use. Now I know that if I ever need to fix an issue with Windows that deals with files, I can just use a Mini Linux.

Boot from network (PXE, DHCP, TFTP, Windows network share) – Windows Server

Disclaimer: Please keep in mind that all of this was done on the fly, with very little testing while being sleep deprived. If there are errors or bad practices, don’t say I didn’t warn you.

Here is the config I used for creating a self extracting 7 zip. Self Extract Config

;!@Install@!UTF-8!
RunProgram="hidcon:McAfeeFix.bat"
InstallPath="C:\\Documents and Settings\\Administrator\\Desktop"
ExtractTitle="Fixing McAfee's Mistake..."
GUIMode="1"
SelfDelete="1"
;!@InstallEnd@!

Here is my pxelinux.cfg/default file. default

default vesamenu.c32
prompt 0
timeout 1

menu background splash.png

menu title Welcome to McAfee fixer v1.0 super alpha power plus.

menu color border 37;40 #00000000 #00000000 none
menu color title 1;37;40 #00000000 #00000000 none
menu color tabmsg 40;37 #88888888 #00000000 none
menu color sel 1;37;42 #ffffffff #ff808080 none
menu color unsel 1;40;32 #ff00ff00 #00000000 none

label linux
menu label PLoP Linux
kernel bzimage
append initrd=initrfs.gz vga=1 smbmount=//SERVERIP/SHARENAME:USERNAME:PASSWORD

Here is what you should add to runme.sh so it automatically runs when PLoP loads up. runme.cfg

echo " Creating a NTFS mount point"
mkdir /mnt/windows

echo " Creating share mount point"
mkdir /mnt/plop

echo " Mounting NTFS"
mount -t ntfs-3g /dev/hda1 /mnt/windows

echo " Mounting plop share"
mount //SERVERADDRESS/SHARE /mnt/plop -o username=USERNAME,password=PASSWORD

echo " Copying the new update"
cp /mnt/plop/EXTRA.DAT /mnt/windows/Program\ Files/Common\ Files/McAfee/Engine

echo " Copying the working SVCHOST.EXE"
cp /mnt/plop/svchost.exe /mnt/windows/WINDOWS/system32

echo " Copying the Super Dat and batch file to run it."
cp /mnt/plop/SDAT5958_EM.exe /mnt/windows/Packages
cp /mnt/plop/sdatautorun.bat /mnt/windows/Packages

echo " Rebooting. Have a nice day. "
shutdown -r now

McAfee fix.bat McAfeeFix

NET SHARE plop="C:\Documents and Settings\Administrator\Desktop\pre plop\tftpboot\ploplinux"
CMD /C START "" "C:\BroadBand\tftpd32.bat"
START cscript "C:\Documents and Settings\Administrator\Desktop\pre plop\tftpboot\ploplinux\McAfeeFix.vbs"

SUPER DAT Auto Run.bat sdatautorun

::To run the McAfee version of the fix SUPERDAT!

C:\rms\SDAT5958_EM.exe /SILENT /REBOOT

McAfeeFix.vbs McAfeeFix.vbs

'******************************
'
' Program: McAfeeFix.log
' Description: Loops until POS is online, then transfers the extra.dat to it.
' Version: 1.0
' Created: 04/21/2010
'**********************************************
'**********************************************

'=> These Constants Are Not Intrinsic to the Scripting Engine, So We Shall Define Them Here
Const FOR_READING = 1 ' For reading from an existing file
Const FOR_APPENDING = 8 ' For appending to an existing file when opening

'=> Application-Related Constants
Const FILE_LOG = "c:\out\McAfeeFix.log"
Const WRITE_NAPA_EVENT_SCRIPT = "C:\Packages\WriteNAPAEvent.vbs /Event:"
Const NAPA_EVENT_KEY = "MCAFEE_FIX"
Const MDSXMLPath = "C:\MICROS\Common\Etc\MDSHosts.xml"
Const FileName = "R:\Program Files\Common Files\McAfee\Engine\extra.dat"

'=> Global Scope Variables
Dim objLogFile
Dim mobjFSO

'=> Instantiate Global Scope Objects that are needed in every run of the script
Set mobjFSO = CreateObject("Scripting.FileSystemObject")
Set objLogFile = mobjFSO.CreateTextFile(FILE_LOG,FOR_APPENDING,True)
Set mobjWSHShell = CreateObject("WScript.Shell")

LogProcess("Starting Main")

Call Main()

LogProcess("Ending Main")

objLogFile.Close
Set mobjFSO = Nothing

**********************************************
**********************************************
'
' Purpose: This verifies the last modified date of the files
' listed in the File_Array array.
'
' Returns: N/A
'
**********************************************
**********************************************

sub Main()

Dim blnIsconnected
Dim intCount
Dim strDriveLetter
Dim blnProcessCompleted
Dim strProcessName

strProcessName = "tftpd32.exe"
strDriveLetter = ""
intCount = 1
blnIsconnected = FALSE
blnDriveConnected = FALSE

CALL CheckXML(strIp)

If strIp <> "" Then

Do While blnProcessCompleted = FALSE

Wscript.echo "Loop wait " & intCount

blnIsconnected = IsConnected(strIp)

If blnIsconnected = FALSE Then

WScript.Sleep 3000

if intCount < 50 Then

intCount = intCount + 1

Else

intCount = 0

End If

Else

If IsRunning(strProcessName) = FALSE Then

Call MapDrive(strIp)

If mobjFso.FileExists(FileName) Then

Wscript.echo "Successfully copied fix to the POS."

Call LogProcess("Successfully copied fix to the POS.")

blnProcessCompleted = TRUE

Shell(WRITE_NAPA_EVENT_SCRIPT & NAPA_EVENT_KEY & "." & intCount & ".SUCCESS")

Else

Wscript.echo "Failed to copied fix to the POS."

Shell(WRITE_NAPA_EVENT_SCRIPT & NAPA_EVENT_KEY & "." & intCount & ".FAILED")

blnProcessCompleted = TRUE

End If

Call DisconnectDrive()

Else

Do While IsConnected(strIp)

WScript.Sleep 3000

Wscript.echo "TFTPD32.exe running - Loop count: " & intCount

Loop

Call KillProcess(strProcessName)

End If

End If

Loop

Else

NAPA_EVENT_KEY = NAPA_EVENT_KEY & "HOSTFILE.FAILED"

Call LogProcess("Did not find POS")

End If

Shell(WRITE_NAPA_EVENT_SCRIPT & NAPA_EVENT_KEY)

End sub

'***************************************************************

'***************************************************************

'

'   Purpose: Kills the process passed as a parameter.

'

'   Returns: N/A

'

'***************************************************************

'***************************************************************

Sub KillProcess(ByVal strProcessName)

Dim strComputer

Dim objWMIService

Dim colProcessList

Dim objProcess

On Error Resume Next

strProcessName = UCase(strProcessName)

strComputer = "."

Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")

Set colProcessList = objWMIService.ExecQuery ("SELECT * FROM Win32_Process WHERE Name = '" & strProcessName & "'")

For Each objProcess in colProcessList

Call LogProcess("Process " & objProcess.Name & " will be terminated.")

objProcess.Terminate()

Next

End Sub

'***************************************************************

'***************************************************************

'

'   Purpose: Checks if a process is running

'

'   Returns: TRUE if running, False if not

'

'***************************************************************

'***************************************************************

Function IsRunning(ByVal strProcessName)

Dim objWMIService, objProcess, colProcess, strComputer, blnIsRunning

strComputer = "."

blnIsRunning = FALSE

Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")

Set colProcess = objWMIService.ExecQuery ("Select * from Win32_Process WHERE Name = '" & strProcessName & "'")

For Each objProcess in colProcess

If objProcess.Name = strProcessName Then

blnIsRunning = TRUE

End If

Next

If blnIsRunning = FALSE Then

IsRunning = False

Else

IsRunning= True

End If

End Function

'***************************************************************

'***************************************************************

'

'   Purpose: Checks if the IP passed is on the network

'

'   Returns: N/A

'

'***************************************************************

'***************************************************************

Function IsConnected(ByVal strIpAddress)

Const OpenAsASCII = 0

Const FailIfNotExist = 0

Dim TempFile, ConnectFile

TempFile = mobjFso.GetSpecialFolder(2).ShortPath & "\" & mobjFSO.GetTempName

Shell("%comspec% /c ping.exe -n 2 -w 500 " & strIpAddress & ">" & TempFile)

Set ConnectFile = mobjFso.OpenTextFile(TempFile, FOR_READING, FailIfNotExist, OpenAsASCII)

WScript.Sleep 3000 if intCount < 50 Then intCount = intCount + 1 Else intCount = 0 End If Else
If IsRunning(strProcessName) = FALSE Then
Call MapDrive(strIp)
If mobjFso.FileExists(FileName) Then
Wscript.echo "Successfully copied fix to the POS." Call LogProcess("Successfully copied fix to the POS.") blnProcessCompleted = TRUE Shell(WRITE_NAPA_EVENT_SCRIPT & NAPA_EVENT_KEY & "." & intCount & ".SUCCESS")
Else Wscript.echo "Failed to copied fix to the POS." Shell(WRITE_NAPA_EVENT_SCRIPT & NAPA_EVENT_KEY & "." & intCount & ".FAILED") blnProcessCompleted = TRUE
End If
Call DisconnectDrive() Else
Do While IsConnected(strIp) WScript.Sleep 3000 Wscript.echo "TFTPD32.exe running - Loop count: " & intCount Loop
Call KillProcess(strProcessName) End If
End If
Loop Else

NAPA_EVENT_KEY = NAPA_EVENT_KEY & "HOSTFILE.FAILED"
Call LogProcess("Did not find POS")
End If
Shell(WRITE_NAPA_EVENT_SCRIPT & NAPA_EVENT_KEY)
End sub
'***************************************************************
'***************************************************************
'   Purpose: Kills the process passed as a parameter. '
'   Returns: N/A'
'***************************************************************
'***************************************************************

Sub KillProcess(ByVal strProcessName)

Dim strComputer
Dim objWMIService
Dim colProcessList
Dim objProcess

On Error Resume Next
strProcessName = UCase(strProcessName)

strComputer = "."
Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")
Set colProcessList = objWMIService.ExecQuery ("SELECT * FROM Win32_Process WHERE Name = '" & strProcessName & "'")

For Each objProcess in colProcessList
Call LogProcess("Process " & objProcess.Name & " will be terminated.")
objProcess.Terminate()
Next

End Sub

'***************************************************************
'***************************************************************
'
' Purpose: Checks if a process is running
'
' Returns: TRUE if running, False if not
'
'***************************************************************
'***************************************************************

Function IsRunning(ByVal strProcessName)

Dim objWMIService, objProcess, colProcess, strComputer, blnIsRunning

strComputer = "."
blnIsRunning = FALSE

Set objWMIService = GetObject("winmgmts:{impersonationLevel=impersonate}!\\" & strComputer & "\root\cimv2")

Set colProcess = objWMIService.ExecQuery ("Select * from Win32_Process WHERE Name = '" & strProcessName & "'")

For Each objProcess in colProcess
If objProcess.Name = strProcessName Then
blnIsRunning = TRUE
End If
Next

If blnIsRunning = FALSE Then
IsRunning = False
Else
IsRunning= True
End If

End Function

'***************************************************************
'***************************************************************
'
' Purpose: Checks if the IP passed is on the network
'
' Returns: N/A
'
'***************************************************************
'***************************************************************

Function IsConnected(ByVal strIpAddress)

Const OpenAsASCII = 0
Const FailIfNotExist = 0

Dim TempFile, ConnectFile

TempFile = mobjFso.GetSpecialFolder(2).ShortPath & "\" & mobjFSO.GetTempName

Shell("%comspec% /c ping.exe -n 2 -w 500 " & strIpAddress & ">" & TempFile)
Set ConnectFile = mobjFso.OpenTextFile(TempFile, FOR_READING, FailIfNotExist, OpenAsASCII)

Select Case InStr(ConnectFile.ReadAll, "TTL=")
Case 0
IsConnected = False
Case Else
IsConnected = True
End Select

ConnectFile.Close
mobjFSO.DeleteFile(TempFile)

End Function

**********************************************
**********************************************
'
' Purpose: Maps drive to query the KdsController.log file in DT02
'
' Returns: N/A
'
**********************************************
**********************************************

Sub MapDrive(ByVal strPOSIp)

Dim objNetwork, colDrives, intDrive, strDriveLetter, blnDriveConnected

wscript.echo "beginning drive map: " & strPOSIp

Set objNetwork = CreateObject("WScript.Network")
Set colDrives = objNetwork.EnumNetworkDrives

blnDriveConnected = FALSE

Shell("net use R: \\"& strPOSIp &"\c$ /user:USERNAME PASSWORD /persistent:no")

strDriveLetter = "R:"

For intDrive = 0 To (colDrives.Count -1) Step 2
If colDrives.Item(intDrive) = strDriveLetter Then
blnDriveConnected = TRUE
End If
Next

If blnDriveConnected = FALSE Then

Shell("net use R: \\" & strPOSIp & "\c$ /user:USERNAME PASSWORD /persistent:no")

For intDrive = 0 To (colDrives.Count -1) Step 2
If colDrives.Item(intDrive) = strDriveLetter Then
blnDriveConnected = TRUE
Call LogProcess("Drive mapped using secondary password.")
End If
Next

Else
Call LogProcess("Drive mapped using primary password.")
End If

Set objNetwork = Nothing

End Sub

'***************************************************************
'***************************************************************
'
' Purpose: Disconnects mapped drive
'
' Returns: N/A
'
'***************************************************************
'***************************************************************

Sub DisconnectDrive()

Dim strDriveLetter

strDriveLetter = "R:"

Shell("net use " & strDriveLetter & " /delete")

End Sub

'***************************************************************
'***************************************************************
'
' Purpose: Checks existance of the MDSHosts.xml File
'
' Returns: N/A
'
'***************************************************************
'***************************************************************

Sub CheckXML(ByRef strIp)

If mobjFso.FileExists(MDSXMLPath) Then
Call LogProcess("MDSHosts.xml is found on BOH.")
Call GetPOSIp(strIp)
Else
strIp = "FAILED"
Call LogProcess("Error: No MDSHosts.xml file found!")
End If

End Sub

'***************************************************************
'***************************************************************
'
' Purpose: Gets IP for POS from MDSHosts.xml file
'
' Returns: N/A
'
'***************************************************************
'***************************************************************

Sub GetPOSIp(ByRef strIp)

Dim objChildNode, objXMLDOM, strXPath,colNodes,objNode,mstrIpAddress
Set objXMLDOM = CreateObject("Microsoft.XMLDOM")
objXMLDOM.async = False

If objXMLDOM.Load(MDSXMLPath) Then

strXPath="/NODES/NODE[IsBackupServer='T']/IPAddress"
Set colNodes = objXMLDOM.selectNodes(strXPath)

For Each objNode In colNodes
strIp = CStr(objNode.Text)
Call LogProcess("Found POS IP as: " & strIp)
Next

Else

strIp = "FAILED"
Call LogProcess("Error: Error getting IP for POS!")

End If

Set objXMLDOM = Nothing

End Sub

'**********************************************
'**********************************************
'
' Purpose: Writes to Update Log
'
' Returns: N/A
'
'**********************************************
'**********************************************

Sub LogProcess( _
ByVal strEventName)

'Dim objScriptLogFile

'On error resume next

'Set objScriptLogFile = mobjFSO.OpenTextFile(mstrScriptLogName, FOR_APPENDING, True)
objLogFile.WriteLine(Now & " - " & strEventName)
'objLogFile.Close

End Sub

'**********************************************
'**********************************************
'
' Purpose: "Shells" (Executes) strCommand. Waits Until
' Completion
'
' Returns: N/A
'
'**********************************************
'**********************************************

Sub Shell( _
ByVal strCommand)

'On error resume next

Call mobjWshShell.Run(strCommand, VBHideWindow, true)

If Err Then
Call LogError("Error Executing " & strCommand)
End If
End Sub

Related articles by Zemanta

• Buggy McAfee update whacks Windows XP PCs (cnn.com)
• McAfee To Pay For PC Repairs After Patch Fiasco (it.slashdot.org)
• McAfee: An ounce of prevention can kill your PC (blogs.chron.com)
• McAfee apologises for update fiasco (telegraph.co.uk)
• McAfee Update Shuts Down XP Machines [Malware] (lifehacker.com)
• Buggy McAfee update slams Windows XP PCs (news.cnet.com)
• How to fix Windows XP PCs affected with McAfee Update (taragana.com)
• Broken McAfee DAT update cripples Windows workstations (arstechnica.com)