How do I correctly bind a Popup to a ToggleButton?

Question

I am trying to do something that seems relatively simple and logic from a user interface level but I have one bug that is very annoying. I have a ToggleButton and I am trying to show a Popup when the button is toggled in and hide the Popup when the button is toggled out. The Popup also hides when the user clicks away from it.

Everything is working as expected with the following XAML except when I click the toggle button after the Popup is shown, the Popup disappears for a split second then reappears.

I suspect what's going on here is that clicking away from the Popup is causing it to toggle the button off then immediately after the button is toggled back on as the mouse clicks it. I just don't know how to go about fixing it.

Any help is appreciated. Thanks.

    <ToggleButton x:Name="TogglePopupButton" Content="My Popup Toggle Button" Width="100" />

    <Popup StaysOpen="False" IsOpen="{Binding IsChecked, ElementName=TogglePopupButton, Mode=TwoWay}">
        <Border Width="100" Height="200" Background="White" BorderThickness="1" BorderBrush="Black">
            <TextBlock>This is a test</TextBlock>
        </Border>                
    </Popup>

Answer 1

Stephans answers has the disadvantage, that the desired behaviour of closing the popup whenever it loses focus also disappears.

I solved it by disabling the toggle-button when the popup is open. An alternative would be to use the IsHitTestVisible Property instead of is enabled:

    <ToggleButton x:Name="TogglePopupButton" Content="My Popup Toggle Button" Width="100"  IsEnabled="{Binding ElementName=ToggledPopup, Path=IsOpen, Converter={StaticResource BoolToInvertedBoolConverter}}"/>
    <Popup x:Name="ToggledPopup" StaysOpen="False" IsOpen="{Binding IsChecked, ElementName=TogglePopupButton, Mode=TwoWay}">
        <Border Width="100" Height="200" Background="White" BorderThickness="1" BorderBrush="Black">
            <TextBlock>This is a test</TextBlock>
        </Border>                
    </Popup>

The converter looks like this:

public class BoolToInvertedBoolConverter : IValueConverter
{
    public object Convert(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
    {
        if (value is bool)
        {
            bool boolValue = (bool)value;
            return !boolValue;
        }
        else
            return false;
    }

    public object ConvertBack(object value, Type targetType, object parameter, System.Globalization.CultureInfo culture)
    {
        throw new NotImplementedException("ConvertBack() of BoolToInvertedBoolConverter is not implemented");
    }
}