Continue reading »]]> I have been working with this issue for a time now, and are waiting for a fix:) Since I have not found any threads or information on this on the web, I thought it could be helpful for others to get the information.

Quick information about the Exchange environment:

We go to sites with Exchange 2010. Each site with two Exchange servers and TMG. One server is running DAG and the other CAS/HT. Both sites are internet-facing, and TMG is publishing in both sites. One is publishing e.g mail.domain.com and the DR site is publishing mail-dr.domain.com. Site A is the primary exchange site, Site B is for DR. All clients connects to the primary site except autodiscover pointing to DR site.

So what is the issue:

When DAG failing over, or if you switch over manually, some random ActiveSync devices will not get redirected to the new active Site. If you manually enter the correct URL the sync start. OWA/Outlook works as expected. When you do an new failover, the same device could be correct redirected. But then some other device could have the problem.

So what is going on:

I first saw the issue when after we applied Exchange 2010 SP3. Before this I the issue did not exist (not that I am aware of). When I first get saw the information, I started to collect information. First of all, I checked the event log on the Exchange servers. And on the site that it was failing from I found this error in Application log on the CAS server:

The Client Access server doesn’t have the InternalURL value set for the Microsoft-Server-ActiveSync virtual directory. This prevents Exchange ServiceDiscovery from finding the MobileSyncService information for user "MBX home server" At least one Client Access server in the user’s mailbox Active Directory site must have the InternalURL value set. The format for the InternalURL value is https://hostname/Microsoft-Server-ActiveSync"

I found out that this event came every time the devices that was unable to redirect tried to sync. But there was not any other events regarding this.

I then checked IIS logs on the site where the DAG was not active (default location C:\inetpub\logs\LogFiles\W3SVC1) and found this.

2013-05-15 02:16:29 IP_CAS_Server OPTIONS /Microsoft-Server-ActiveSync/default.eas &Log=RdirTo:https%3a%2f%2fPrimarysiteURL%2fMicrosoft-Server-ActiveSync_V0_LdapC1_LdapL109_Cpo19890_Fet19999_S130_Error:DiscoveryInfoMissing_Mbx:FQDN_MBXServer_Budget:(D)Conn%3a1%2cHangingConn%3a0%2cAD%3a%24null%2f%24null%2f1%25%2cCAS%3a%24null%2f%24null%2f0%25%2cAB%3a%24null%2f%24null%2f0%25%2cRPC%3a%24null%2f%24null%2f0%25%2cFC%3a1000%2f0%2cPolicy%3aDefaultThrottlingPolicy%5F221b878b-f351-48f1-b524-2ae00cda8947%2cNorm_ 443 domain\username IP-address from TMG publishing Apple-iPhone4C1/1002.329 403 0 0 19999

So what does this mean:

1. We can see that the ActiveSync virtual directory is trying to redirect to the primary site http://primarysiteurl

2. The mailbox server name in primary site is FQDN_MBXServer in primarysite

3. The redirect failed: the error information is ‘DiscoveryInfoMissing’ and the error code is ‘403’ instead of the 451 redirect it should have got.

I have not found any more information on this, so I opened a case with Microsoft. And this was the answer I got.

Then I do some researches on our situation, after applying the SP3, when we manually switch the DAG over back to primary site again, the ActiveSync device cannot automatically redirect to the primary site, and get a 451 redirect, instead getting a 403 error. And it is a known issue, we have reported this problem to our product group. It will be fixed in the next generation SP3 RU2. Then let’s wait the update patiently. Hope you can understand.

I will continue work with support to hopefully get a fix for this before RU2.

Continue reading »]]> Hi all!

Needed to sign a powershell script today, an thought it could be of interest for others as well. So here are the steps.

First of all, you need to request/obtain a Code-signing certificate. I usually do this by creating a own template, request a certificate and use GPO to distribute this to Trusted Publisher. This is not covered by this post, but it is pretty straight forward.

So, back to signing scripts. If you get this error message when trying to run a script (script cannot be loaded. The ps1. fie is not digitally signed), many just changing the executionpolicy to be unrestricted… It is fast, but might not  be the best solution:)

It is easy to sign a script when you got the basics in place (code-signing certificate and the certificate distributed to the clients Trusted publisher certificate store).

On the machine you want to sign the script from, you need to install the Code-Signing certificate on either Computer or User. I use computer in this example. To make sure you got the CodeSigning certificate run the following command:

Get-ChildItem cert:\LocalMachine\my –CodeSigningCert

This command will list you Code signing certificate installed on local computer

Then run this command to get the certificate in a variable called $cert.

$cert = Get-ChildItem cert:\LocalMachine\my –CodeSigningCert

Then sign the script with command:

Set-AuthenticodeSignature -Certificate $cert -FilePath C:\Install\Get-DAGHealth.ps1

As you can see in the bottom of the script, there has been added a signature block.

Remember, if you do a change on the script now. It needs to be signed again:)

Happy signing!

Continue reading »]]> I recently tested some restore operations on Exchange 2013. This is the step I did to do a “single mail” restore:

1. Restore your  backup mailbox database and logs to alternative location. (But do not change the filename on the DB, if you do this eseutil is not able to get this in clean shutdown state).
2. Because this is is an restored database, the database most likely will in “dirty shutdown” status. Here is how I got  it in “Clean Shutdown” with esutil

Verify that the restored database has status “dirty shutdown” with eseutil /mh E:\testrestore\EX13.MDB02.edb  

Also look at the Log required (4-4) and log committed (0-5). These log files are needed at a minimum to recover the database to a clean shutdown without data loss. In my case i have both this files

You can check the consistency of the needed logs by using  the command eseutil /ml together with your log directory path and log prefix, in my example E02: eseutl /ml F:\Testrestore\RDB1\E02

When I did repair on the database I had huge problems with finding the correct eseutil command. But after I found this blogpost: http://blogs.technet.com/b/mspfe/archive/2012/09/06/why-exchange-databases-might-remain-dirty-after-eseutil-r-recovery.aspx  I found out that I could use the command eseutil /R E02 /l “F:\Testrestore\RDB1" /s "F:\Testrestore\RDB1" /d "E:\Testrestore"

Running this modified command, everything worked and the database ended up in a “clean shutdown”

It’s time to create a recovery database. Use the shell to create a recovery database and point it to the database we want to recover data from. In my example this is

New-MailboxDatabase -Recovery -Name TestRestore -Server Servername -EdbFilePath "E:\Testrestore\EX13.MDB02.EDB" –LogFolderPath "F:\Testrestore\RDB1"

Mount the database: Mount-Database testrestore.

Now it is time to do the fun work. Lets do some restore.

I will in this example restore the whole mailbox to a subfolder named Restore. To create a restore request, you must provide the DisplayName, LegacyDN, or MailboxGUID for the soft-deleted or disabled mailbox. To get a list of mailboxes you can restore run Get-mailboxStatistics –Database Testrestore

When I run the command documented on TechNet I got a strange error: Error detail: must be logging in with GUIDs. not legDN. I tried with the DisplayName, LegacyDN and MailboxGUID without success.

New-MaiboxRestoreRequest -SourceDatabase Testrestore-SourceStoreMailbox "EX2013" -TargetMailbox UPN –Targetfolder Restore

But I found another parameter I could use: -SkipMerging StorageProviderForSource. And guess what….

I worked:)

Continue reading »]]> Before I installed Exchange 2013 I issued a certificate from the internal CA server. Everything looked fine. The installation went fine without issues. But when I tried to administrate Exchange 2013 from the ECP and OWA web I got the following error: Invalid provider type specified.

I also saw the details “System.Security.Cryptography…. And guessed it had to be something with the certificate i issued before the installation

Earlier I have had issues with system and certificate using the “new” Microsoft Software Key Storage Provider using cryptographic API Cryptography Next Generation (CNG). The fix then has been to use a version 2 template (windows 2003) which use the Microsoft RSA SChannel Cryptographic Provider. This templates use the old CryptoAPI, which makes use of providers called cryptographic service providers (CSPs)

When I checked the certificate used in Exchange 2013 using certutil –store my, I verified that it was using Microsoft Software Key Storage Provider

I could also verify this when I requested the certificate from MMC

Using V3 template:

Using a V2 template

I therefore created a new Web Server V2 template in Microsoft Certification Authority and requested a new Web certificate on the Exchange 2013 server. And assigned all the Exchange services to the new certicate using Enable-ExchangeCertificate command. Restarted the server (it actually needed restart). And voila finally I could manage my Exchange server.

Come on Microsoft!!

Continue reading »]]> When I tried to install the first Exchange 2013 CU1 server in coexistence with Exchange 2010 SP3 (yes I had updated all Exchange 2010 servers to SP3), I got the following message:

All Exchange 2010 servers in the organization must have Exchange 2010 Service Pack 3 or later installed. The following servers don’t meet this requirement: EdgeServerName
For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.E15E14CoexistenceMinMajorVersionRequirement.aspx

This was the Edge server, and I checked that this server actually had SP3 installed.

Since it was referringt to the Edge server I tried to create a new Edge subscription. After I was able to continue with the installation:)

Cheers!

Continue reading »]]> It’s been a great year since I received the Dell TechCenter Rockstar title last year. Thanks to everybody at Dell that was a part of it. Yesterday was the day were we would now who got their title renewed and the new that would be nominated. It was great to see the list on the Dell TechCenter blog and see that my name was there. I’m honored to be given the title for one more year and looking forward to working with my fellow Rockstars and the Dell TechCenter team

.

Continue reading »]]> A customer got a issue with Forefront EP 2010. It was installed on their Terminal Servers, and a file was detected by Forefront. The “biggest” problem was that all users got a pop-up telling that the server needed to be restarted to remove the virus. And if one user clicked restart, it would restart, even if they do not have permissons to do this. Not what you want done without control or info to the other users logged on. The fix for this is to add a registry setting that prevents access to the user Interface of Forefront on the terminal servers. Found this is on the Microsoft forums.

HKLM\SOFTWARE\Policies\Microsoft\Microsoft Antimalware\UX Configuration

REG_DWORD – UILockdown = 1

Add this key, and the pop-up is not shown anymore.

This is the link to the forum post: http://social.technet.microsoft.com/Forums/en-US/FCSNext/thread/d28331a0-5c91-4786-9d52-e78b966f4d78

Continue reading »]]> Yesterday Microsoft released a hotfix that’s a big collection of fixes for slow boot and slow login on Windows 7 and Windows Server 2008 R2. Check this article from one of the PFE’s: http://blogs.technet.com/b/askpfeplat/archive/2013/03/12/slow-boot-slow-login-sbsl-hotfix-rollup-for-windows-7-and-server-2008-r2-available-today.aspx

Tested this on a computer that before the hotfix would use 20+ minutes, after applying the hotfix it now takes about 1-1,5 minute to start and logon. So quite a big improvement

Continue reading »]]> Ever wondered how rate limits on Distribution Points in ConfigMgr 2012 work? When I one day accidentally updated all the distribution points at a customer site it was time to learn quickly.

There are three modes for rate limits and these are set per Distribution Point. Unlimited sending is self-explainatory, but the pulse mode and the option to limit specific maximum transfer rates per hour are useful in different scenarios.

I´ll explain the limited to specified maximum transfer rates by hour option first, since it was what helped me quickly in my scenario. First of all, there is no native ConfigMgr support for BITS when distributing content to Distribution Points, therefore, ConfigMgr does not monitor the traffic and/or the amount of data going over the link. Instead, it uses the time available for sending and divides the time into time slots where it sends 100% of data. So, for instance, if you use the option to use only 50% of bandwidth during work hours (ie 8AM to 4PM), it will divide the amount of available time into time slots and send 100% of data 50% of the time, and send 0% data the other 50% of the time. This is useful if you don´t want to clog the available bandwidth on the network during work hours or if you need to limit the transfers for some other reason. This can be set per hour for every day, but unfortunately you can´t set it for specific days (as you sort of could in Configuration Manager 2007).

It is also possible to use pulse mode, in which case you can specify the size of the data blocks sent by the site system to the distribution point and a delay in seconds for how often these blocks should be sent. This is useful if you have slow links to your distribution points, but it requires you to calculate how much data is sent. If you set it too low your packages will never be distributed, if you set it too high it will still clog your bandwidth. Notice that these are radio buttons, so you can only choose one option at a time for one specific Distribution Point.

As you can see in my example I set it to transfer only 50% during working hours for all my distribution points, and I immediately saw the effect on my site servers bandwidth.

It is also possible to use the Schedule option on your Distribution Point to restrict what kind of traffic can be sent during different hours of the day. This applies to the priority of content, so say that you have software update packages that you know you want to distribute but everything else can wait. If you set your software update packages to high priority and everything else at medium, then you can effectively block medium priority content from being distributed but let high priority content get distributed by selecting the option Allow high priority only.

You can read more about the different distribution point settings in the Configuration Manager Document Library.

Continue reading »]]> It has been a while since I wrote a good blog post. So the time has come for me to write a little about Windows security from a Penetration tester perspective. Hope you like it.

There is probably a tons of guides and blog posts on how to hardening your Windows clients. In this post however I will focus on what a Penetration tester looks for and tests. Remember that a Penetration tester often wants to test how far he can get in a network, but a real hacker often stops when he gets what he wants. A hacker can often be satisfied by getting a shell running as the logged on user.

I always hear as long as you get physical access to the machine you own it. And that is somehow true, but you still need to hack it to get in. So this will take this statement to a test.

In this post, the goal of the Penetration tester is to get as far as he can get. The ultimate goal is domain admin on a domain controller. This is of course often scenario based for a penetration tester. So I will describe 2 scenarios and how I tested them, then I will explain mitigation factors to prevent this in the future.

Scenario 1 is that I somehow get my hands on a computer inside a organization. (breakin ,social engineering etc.)
- This computer is not bitlocker encrypted.
- The domain user using this machine is not a local admin.
- The Local administrator account is disabled.
- The machine has all normal physical ports – USB,PCMCIA,HDMI,Firewire etc.
- The Windows Firewall is configured opened for 445, ICMP and WinRM from all hosts within the domain.

Scenario 2 is also a computer that I get my hands on inside a organization. (breakin ,social engineering etc.)
- The computer is bitlocker encrypted without PIN.
- The domain user that uses this machine is not a local administrator.
- The Local administrator account is enabled.
- There was also a domain\helpdesk user cached on the machine
- The machine has all normal physical ports – USB,PCMCIA,HDMI,Firewire etc
- The Windows Firewall is configured opened for 445, ICMP and WinRM from all hosts within the domain.

The big difference in these 2 scenarios is the fact that scenario 1 includes bitlocker encryption and that scenario 2 has cached credentials for a helpdesk account. Okay, lets start the walkthrough of the scenarios.

Scenario 1:
First goal is to get local admin access. How can we do that?? In many ways basically. The easiest way I know of is to change binaries. Let me explain.
In Windows there is almost an EXE/DLL file behind every thing you click on. For example the “ease of access” or “Sticky Keys” (5 times on SHIFT). There is also the Windows+P to change display that launces a EXE file (displayswitch.exe). I will not go into detail on how to find out what EXE files that is used on different action, but the key tool is Process monitor by Mark Russinovich.

The exe file behind “ease of access” is named utilman.exe and “Sticky keys” is named sethc.exe. So what I want to do is to change these binaries with cmd.exe. How do I do this? I boot up on a Windows Installation Media and hit SHIFT+F10. (You can of course use any boot cd you want)

This will launch a command prompt from where I can Issue the following command:
copy c:\windows\system32\cmd.exe c:\windows\system32\utilman.exe (answer Y to confirm the overwrite)
copy c:\windows\system32\cmd.exe c:\windows\system32\sethc.exe (answer Y to confirm the overwrite)

Then we are done with the binary inject attack. Boot up the computer as normal and hit either the “ease of access” button or use 5x Shift key to launch “sticky keys”.

So as you can see on the screenshot you can now get a shell in 2 ways ( I could do the same with displayswitch.exe and using Windows+P to get a shell). Each resulting in you running shell as system. So whats next. Thats right, create accounts…..
Run the following command in either of the shells:
net user msitpros H4ck3r4L1fe /add
net localgroup administrators msitpros /add

Then you need to reboot the computer in order for the new account to be active. After the boot we can log on as .\msitpros and password H4ck3r4L1fe and we are now a local admin. This was I see it phase 1 in the test. Next phase is finding NTLM hashes on the box in order to user the Pass-the-hash attack so we can harvest other hashes.

So how to we dump the NTLM hashes? Well, there are several ways of doing this. My “new” favorite way of doing this is by using a Powershell script(what else??). In order to dump hashes from Powershell we need to run it as the SYSTEM account. So how do we do that excactly? Easy, download a copy of pstools from Mr. Mark Russinovich (great guy). It can be found here: http://technet.microsoft.com/en-us/sysinternals/bb896649.aspx
Extract it and run the following command from within the PSTools folder:
.\psexec -i -s powershell.exe

This should then launch a powershell command window. You can now run the whoami command to verify your identity.

Next up is executing this well written script :
Source: http://truesec.com/Content/SiteContent/Tools/Get-TSLocalHash.txt (Kudos to Truesec)
This will show you the NTLM hashes that is present on the machine.

And getting the hashes is like getting access to the users password. So, lets try to use them for something fun, maybe connect to another machine for instance.
For my next “trick” I am using a tool called “Windows Credential Editor” or WCE. This tool can be downloaded from here:
X86: http://www.ampliasecurity.com/research/wce_v1_3beta.tgz
X64: http://www.ampliasecurity.com/research/wce_v1_3beta_x64.tgz
(In order to extract this using 7-zip you need to extract the extracted file once more)

Okay, so you got the tool in place. Let’s use it to pass the hash to another machine. Let take the Hash to “Tro Jan” and pass it to another box to see if it works. This is the syntax for launching another command using this hash:
wce.exe -s <user>:<domain>:<LM hash>:<NT hash> -c cmd.exe
or with values:
wce -s “Tro Jan”:Win7-Target:aad3b435b51404eeaad3b435b51404ee:e19ccf75ee54e06b06a5907af13cef42 -c cmd.exe
‘ note that Win7-Target must be the machine name of the target machine when using local accounts, otherwise use domain name.

This should now launch another cmd. And when you try to use for example “net use \\win7-target\c$” from the new cmd it will not ask you for username or password. It sends your crafted Hash to the other machine. (Evil laughter time – Mohahaha)
So now I just have to bounce around the machines until I find one with a domain admin account. I will then use the same method to access the domain controllers and then gain the ultimate goal.

Scenario 2:

This computer has as you remember bitlocker, fortunate for me it does not have a startup pin. So how far can I get in this scenario? I got no username or password. If I try to use a offline media I will not get into the disk, because of the bitlocker encryption. So a cool hack is to use the firewire exploit. I know this might be very old information to many of you, but as long as it is still valid I choose to use it and describe it. If you have never heard of anything like a firewire exploit, then let me explain it to you. Firewire is a technology that get’s direct access to the memory of the machine (its a feature, not a bug….). It is designed to have very high speed, and in order to achieve this they found out that direct memory access was the way to go.

Okay, so then I use Inception ( a cool tool that can bypass winlogon with firewire by patching MSv1_0.dll in memory) in order to hack my way into the local admin account.

*source: http://www.breaknenter.org/projects/inception/

If you want to see the attack and how it works take a look at this video:
http://www.youtube.com/watch?v=tsb6ojxa3ik

Fortunate for me the administrator account is enabled. So I will just launch the attack from Inception and be logged on as administrator, and the create another user with local administrator rights.
I will then logg on with my new account and use Pass-the-hash attack as described in scenario 1 with the helpdesk account as described in the scenario overview. I will do this until I find a domain admin account and then move onto the domain controllers. And there we are….King of the castle…..
Another approach could be to crack the hash to the helpdesk account using rainbow tables. (more on that in a later post)

Mitigation:

So what is the lesson learned here? First of all, it is important to think about security at all places, not just only the servers.

In order for bitlocker to have an effect you must at least disable Firewire. And not just disable it in BIOS. If you just disable the Firewire port itself you will not prevent anyone from using a PCMCIA card with Firewire. What I am trying to say is that we need to restrict the setup class GUID. Microsoft has described this problem and the solution here: http://support.microsoft.com/kb/2516445/en-us

- Bitlocker would have helped a lot in scenario 1, but not without disabling Firewire.
- Disabling local adminsitrator accounts would have helped a lot in scenario 2.
- Blocking communication between clients would have stopped the Pass-The-Hash against other workstation and not letting me to harvest hash’es
- If both scenarios had PIN on the bitlocker I would get nowhere. So a good mitigation factor is to implement bitlocker with PIN and not just TPM.
- After enabling PIN on bitlocker I would also make sure that the machines are going to Hibernation rather than standby.

To sum it all up: disable local admin accounts, do not give anyone local admin rights, disable firewire, enable bitlocker with TPM and PIN, block client-to-client communication and remove debug priviliges

In my next post I hopefully will write about how to escalate privileges using different techniques and of course how to prevent them.