Jump to content
tlphipps

File BITS Download - Working Example

Recommended Posts

I've been fighting for some time with trying to download large files (500MB+) reliably to Automate agents. I've created retry loops, etc. but I never liked the overall results.

So I decided to try using the BITS download functions in CWA to see if this could make my downloads more reliable. Then I hit the stumbling block of really poor documentation provided by CW for this function set and process. After several hours of reading, testing, bouncing ideas off others, and doing my own Powershell exploration and hacking, I FINALLY have a working proof-of-concept code that I'm sharing here. There's not a lot of error checking/handling in this. I'd like to add basic file size checks or even file hash checks, but I was just so excited when I finally got THIS working that I wanted to share it in case it might help others in the future.

There are actually 2 scripts involved but they'll both be imported from the single XML attached here. BITS Download - ERGTEST is the primary script and where you'll want to edit the ISO filename and possibly other variables. The second script, BITS Status - ERGTEST is really just a helper script that provides script log output showing the download status and performing the final "completion" task that takes the downloaded temp file and turns it into the real deal.

I think the variable names are pretty self-explanatory and obvious. But if you have any questions, I'm happy to share the knowledge I've learned during this process.

NOTE: Make sure you change line 2 (ISO_Filename) to the full URL for your download. And also update line 5 (FinalFileLength) to the correct size (in bytes) for your file.

BITS Download - ERGTEST.xml

Edited by tlphipps
  • Thanks 1

Share this post


Link to post
Share on other sites

Let me see if I can do that succinctly....

BITS is a background task by design. So the initial CWA function puts a job in the BITS queue and the BITS service on the local agent then picks up the job from the queue and starts the download process. The great thing about BITS is that it will run with lower priority to avoid hogging all your bandwidth, and it automatically resumes the download if something pauses it (computer goes to sleep, or even a shutdown/reboot).

So once the task is running in the background, you have to keep up with the status of your download to know when it actually finishes. This is where CWA documentation gets really bad. My implementation calls the second (status) script immediately after queueing the BITS job. It then uses the BITS status function to query the status of the download job from the BITS engine and gives you a log entry showing the percent complete (based on the data from the BITS engine). There's a continuous loop that runs as long as the BITS job status doesn't show 'TRANSFERRED.' Once that status is reached, something has to tell the BITS engine to 'complete' the job (rename the temp file it created to the final destination filename). Again, CWA kinda lets you down here. I would expect it to automatically handle this, but alas, that's not the behavior I'm seeing. So after the status of 'TRANSFERRED' is found, I execute a powershell command to 'complete' the BITS job which causes the BITS engine to rename the output file.

And voila, you have a file download. Overall it's not too bad, but lacking CWA documentation makes it really hard to piece this together. Especially when said documentation references example scripts that I could never find anywhere. They seem to be from an old workshop or something.

Oh, and based on feedback I got from others about BITS transfers causing them issues previously, I borrowed some code from @Gavsto and added an automatic fall-back method for downloading directly with PowerShell as well in case the original BITS download doesn't work for some reason. My loop logic for status updates in that section isn't working for some reason, but I went ahead and posted up what I have above as it greatly enhances what I had in v1.

Share this post


Link to post
Share on other sites

The BITS function allows you to avoid all this extra work by specifying a trigger script that is called when the transfer completes. So, the first time you call the BITS function, your script should exit. When the download is finished, your selected script will be restarted. Just test for the file to Exist, and perhaps perform an MD5 check, and then continue. If the file DOESNT Exist, try the BITS Starus function. If the download is still in process, just exit. If the download has failed, you could retry or attempt a direct download. The background nature/design of a BITS transfer makes the looping/checking/waiting unnecessary. If you are going to babysit the transfer, then don’t ask BITS to manage it. If you are going to use BITS, then don’t try and do its job. For assurance that BITS has started, you could queue your script to trigger 10 minutes after the first exit, just to get a BITS status and confirm it has started. After that, just wait for your script to be called back up. 

I’ll work on posting a working example that uses the BITS function successfully.

Share this post


Link to post
Share on other sites

Yeah, I read that's how it SHOULD work in the documentation, but my testing didn't show that "automatic script call/restart" working. And in my testing, I HAVE found agents where BITS just won't work even though it queues up the job (the job immediately goes to a transient error state). Your point about not monitoring something that's designed for background running is certainly valid, but in my org, techs/engineers are impatient and clueless about how these scripts really work. So if they don't see something 'running' in the script tab and have some relatively recent status update listed, they freak out and start causing problems trying to re-run scripts, etc. Maybe that's just here.

I also found another set of documentation that specifically says you SHOULD call the status script right after starting the download (which of course contradicts what they say in most other places about this). And since I wasn't seeing the other method work for me, I went this route.

I would absolutely LOVE to see a working example that matches what the documentation implies (automatic call after completion of download).

Share this post


Link to post
Share on other sites

So I was able to utilize BITS different way (avoiding the LT function) and just using "Execute Script>Powershell (bypass)" instead.  This way, it waits for that powershell command to complete before moving on to the next script step and you don't have to do all that file checking crap (although an MD5 check against the source is still a good idea)

start-bitstransfer -source http://server.com/Windows.iso -Destination @pathToISO@/@isoName@.iso

Share this post


Link to post
Share on other sites

Sometimes I use PowerShell to download and install. 

You will notice that I used `Start-Process -Wait` so it will complete before going to the next entry.  

& {
# PowerShell Download & Install Google Chrome
$SoftwarePath = "C:\Support\Google"
$DownloadPath = "https://dl.google.com/chrome/install/GoogleChromeStandaloneEnterprise64.msi"
    $Filename = [System.IO.Path]::GetFileName($DownloadPath)
    $SoftwareFullPath = "$($SoftwarePath)\$Filename"
    $wc = New-Object System.Net.WebClient
    if (!(Test-Path $SoftwarePath)) {md $SoftwarePath}
    Set-Location $SoftwarePath
    if ((Test-Path $SoftwareFullPath)) {Remove-Item $SoftwareFullPath}
    $wc.DownloadFile($DownloadPath, $SoftwareFullPath)
(Start-Process "msiexec.exe" -ArgumentList "/i $($SoftwareFullPath) /qn" -NoNewWindow -Wait -PassThru).ExitCode
  }


& {
# PowerShell Download & Install - ConnectWise Manage
$SoftwarePath = "C:\Support\ConnectWise"
$DownloadPath = "https://university.connectwise.com/install/2019.4/ConnectWise-Manage-Internet-Client.msi"
    $Filename = [System.IO.Path]::GetFileName($DownloadPath)
    $SoftwareFullPath = "$($SoftwarePath)\$Filename"
    $wc = New-Object System.Net.WebClient
    if (!(Test-Path $SoftwarePath)) {md $SoftwarePath}
    Set-Location $SoftwarePath
    if ((Test-Path $SoftwareFullPath)) {Remove-Item $SoftwareFullPath}
    $wc.DownloadFile($DownloadPath, $SoftwareFullPath)
(Start-Process "msiexec.exe" -ArgumentList "/i $($SoftwareFullPath) /qn" -NoNewWindow -Wait -PassThru).ExitCode 
  }

 

Edited by Braingears

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...