How to make WPF programs better adapt to UI automation

UI Automation has been built into Windows for a long time (UIAutomation has been provided since Windows XP SP3), and UI Automation support has been provided since the first version of WPF. So it stands to reason that if you use WPF, your UI is ready to be automated at any time.

Researcher at Microsoft MVP Labs

While WPF support is good, I have a few points to mention:

1. Here I am talking about "UI automation", not "UI automation testing"; the former has a wider scope than the latter, because the former can be used not only for UI automation testing, but also for screen-reading software. people for convenience.
2. WPF provides UI automation support from the mechanism level, but it can't stand many people who don't know the relevant mechanism accidentally change it, so it is necessary to talk about it in this article.
   Next, I will talk about the following aspects, only talking about the use level, not going deep into the principle level.

UI Automation that comes with WPF

In order to facilitate the demonstration, I use the template that comes with Visual Studio to create a default WPF application, I will constantly modify this program, and then use the UI automation test software I wrote myself to verify its automatic adaptation effect.

▍Which controls come with complete UI automation

on Windows

UIAutomation control name	Corresponding WPF control name	translate
button	Button	button
calendar	Calendar	calendar
checkbox	CheckBox	check box
combobox	ComboBox	Combo Box
custom	UserControl	custom control
datagrid	DataGrid	data sheet
dataitem	DataItem	data table entry
document		Documentation
edit	TextBox	text box
group		combination
header		title
headeritem		title item
hyperlink		Hyperlink
image	Image	image
list	ListBox	list
listitem	ListBoxItem	list item
menu	Menu	menu
menuitem	MenuItem	Menu Item
menubar		Menu Bar
pane		container
progressbar	ProgressBar	progress bar
radiobutton	RadioButton	Single box
scrollbar	ScrollBar	rolling tone
separator	Separator	delimiter
slider	Slider	slider
spinner		spinner
splitbutton		split button
statusbar	StatusBar	Status Bar
tab	TabControl	Tab
tabitem	TabItem	tab item
table		sheet
text	TextBlock	text
thumb	Thumb
titlebar		title
toolbar	ToolBar	toolbar
tooltip	ToolTip	tooltip
tree	TreeView	tree view
treeitem	TreeViewItem	tree view item
window	Window	window

In addition, there are two other new control types that support UI automation in new Windows systems (or in UWP/WinUI programs):

UIAutomation control name	Corresponding WPF control name	translate
semanticzoom	SemanticZoom
appbar	AppBar

However, from the actual test situation, Microsoft has no longer used these two special controls, but replaced these two special controls with the combination of the previous commonly used controls.

▍The support of WPF's own controls

In order to visually see the support of each built-in control of WPF for UI automation, I added various common controls to the WPF program I just created, and then used the UI automation test software I wrote to capture this window.

It can be found that WPF's own controls correctly expose various required controls for UI automation. At the very least, screen-reading software for the blind can accurately read the textual descriptions of all controls.

Below is a video of this intuitive capture

Specifically, WPF has these features by default:

1. All interactable controls can be captured as a whole, and each interactable part can also be captured separately (eg calendar and inner buttons, tree and inner items, scroll bars and inner buttons, etc.).
2. The parts of the text that change in the control are also properly exposed to UI automation (eg text inside buttons, list item text, menu items, etc.).
3. Container and layout class controls are not exposed to UI automation (eg Grid, StackPanel, Border, etc., do not appear in automated tests).
4. UserControls are exposed to UI Automation.

▍Correspondence between WPF properties and UI Automation properties by default

Maybe someone knows that WPF has a set of automation-related APIs to adapt to UI automation. Is a set of additional properties, such as UIAutomationProperties.Xxx. for example:

• AutomationProperties.AutomationId
• AutomationProperties.Name
• there are more……

But when we write controls, we don't actually need to actively and directly set these properties. While no values are set for these additional properties, other useful properties have been exposed when exposing related properties to UI automation.
for example:

• If you set the control's name x:Name="WalterlvDemoButton", then after UI Automation captures this control, its automation Id is WalterlvDemoButton.
• If you set the content of the control (such as the Content of a button/check box/radio box/list item, such as the Header of a menu item/tab), then after UI Automation captures this control, its Automation Name is the corresponding specified of these properties.
• And even if you don't have any settings, the automation class name is the class name of the control, IsEnabled corresponds to the IsEnabled of the control itself, and IsVisible also corresponds to the IsVisible of the control itself.

With the above features as a guarantee, make good use of these built-in controls, and correctly follow the original attribute meaning of the controls when making the control layout and adjusting the style. There is no need to do any adaptations specifically for UI automation. match. However, this is not the case in reality...

What breaks UI automation in WPF

Many times, when we write code, we may pay too much attention to what the final result looks like, and ignore the original hierarchy and attribute meaning of the controls, which may lead to the control and hierarchy of our program exposed to UI automation testing. Weird, not even readable.

Below, I list a few examples:

1. Originally, the content attribute was used to set the text attribute of the button, but one day I wanted to make a very special style, and the text was written in the template (Template) alone, and the content attribute of the button was not directly set. In this way, when the UI automation software grabs this button, it does not know what function the button is, and will grab a button without a text description.
2. The list or tree is bound to a source (ItemsSource), and each item in the source collection is an item in the ViewModel (such as Walterlv.Demo.DemoItem type), this type does not override the ToString method, so the list item is exposed The names given to UI Automation will be repeated meaningless strings (eg both Walterlv.Demo.DemoItem).
3. Some buttons or list items do not have any textual descriptions, they are controls made entirely of images. If the button doesn't have a name assigned, it's indistinguishable from any other button of its kind; a list-like control basically doesn't expose any useful information in this case.
4. Some controls obviously want to be interactive, but they prefer to use layout or decorative controls such as Grid and Border for styling, and finally use general events such as MouseDown for interaction. This is basically the same as dropping support for all UI automation with its own controls.
5. Make very complex interactive controls yourself (such as making a canvas yourself), which inherits from the very low-level FrameworkElement. Although this control specifies the control style and template, it already exposes no useful information for UI automation.

The latter two, 4 and 5, UI Automation can't even capture such controls. After all, WPF is not very good at exposing all controls to UI automation by default. Otherwise, UI automation testing software or screen reading software will face a complex and huge UI automation tree like the WPF visual tree.

Best Practices for UI Automation in WPF

After understanding the existing characteristics of WPF UI automation, we will break down the above pit points one by one, which is the best practice we recommend.

1. Try to keep the UI automation mechanism that comes with WPF, and avoid overly complex customization of styles and templates. If you want to do it, you should use ready-made common properties as much as possible, instead of defining new properties yourself (for example, use Content instead of defining a new one). TitleText and the like).
2. If a collection of ViewModels will be bound to a UI list or tree, the ViewModel should override the ToString() method to return useful information that is readable by the user (don't print all properties at once like console output).
3. If a button or image doesn't have any textual description, set the x:Name property to add a unique Id; better, set the AutomationProperties.Name attached property to specify a friendly name for the visually impaired.
4. If a button or image without a text description is in the list, set the AutomationProperties.Id property for it to bind a unique Id that distinguishes each other, and then set the AutomationProperties.Name attached property to specify a friendly name for the visually impaired to read.
5. Try to use common controls to do the interaction corresponding to the controls (for example, use buttons like a button, and use a combo box like a combo box), instead of using Grid, Border and other controls for layout or decoration to handle at will.
6. If you must do a special control interaction (no existing control can represent this interaction), then make full use of the user control (UserControl) will automatically expose the characteristics of UI automation to make a user control. Conversely, if you are using a user control just to split the code, you should override the OnCreateAutomationPeer method for the control, returning null to avoid the control appearing in the UI automation hierarchy.
7. If you also want a special interactive control to be reused (not suitable for UserControl), then you need to override the OnCreateAutomationPeer method for this control, returning an instance of a suitable AutomationPeer.

 // 对于上述第 6 点，应该为用户控件重写此方法。
protected override AutomationPeer? OnCreateAutomationPeer()
{    
return null;
}
public class WalterlvDemoControl : FrameworkElement
{    
// 对于上述第 7 点，应该为用户控件重写此方法。    
protected override AutomationPeer? OnCreateAutomationPeer()    
{        
return new WalterlvDemoAutomationPeer(this);    
}
}
// 自定义的 AutomationPeer。只需要继承自 FrameworkElementAutomationPeer 就可自动拥有大量现成自动化属性的支持。
public class WalterlvDemoAutomationPeer : FrameworkElementAutomationPeer
{    
public WalterlvDemoAutomationPeer(WalterlvDemoControl demo) : base(demo)    
{    
}    
// 在 AutomationControlType 里找一个最能反应你所写控件交互类型的类型，    
// 准确返回类型可以让 UI 自动化软件针对性地做一些自动化操作（例如按钮的点击），    
// 如果找不到类似的就说明是全新种类的控件，应返回 Custom。    protected override AutomationControlType GetAutomationControlTypeCore()    
{        
return AutomationControlType.Custom;    
}    
// 针对上面返回的类型，这里给一个本地化的控件类型名。    
protected override string GetLocalizedControlTypeCore()    
{        
return "吕毅的示例控件";    
}    
// 这里的文字就类似于按钮的 Content 属性一样，是给用户“看”的，可被读屏软件读出。    // 你可以考虑返回你某个自定义属性的值或某些自定义属性组合的值，而这个值最能向用户反映此控件当前的状态。    
protected override string GetNameCore()    
{        
return "吕毅在 https://blog.walterlv.com 中展示的博客文本。";    
}
}

An example of UI automation adaptation to a ListBox that is almost entirely composed of images. In the following animation, if there is no adaptation at all, then only the name of the ViewModel that is completely indistinguishable will be obtained when capturing, which is also the class name Walterlv.Demo.ThemeItem generated by ToString by default.

References

UI Automation - Win32 apps - Microsoft Docs:https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32?ocid=AID3052907
UI Automation Overview - .NET Framework - Microsoft Docs: https://docs.microsoft.com/en-us/dotnet/framework/ui-automation/ui-automation-overview?ocid=AID3052907

Microsoft Most Valuable Professional (MVP)

The Microsoft Most Valuable Professional is a global award given to third-party technology professionals by Microsoft Corporation. For 29 years, technology community leaders around the world have received this award for sharing their expertise and experience in technology communities both online and offline.
MVPs are a carefully selected team of experts who represent the most skilled and intelligent minds, passionate and helpful experts who are deeply invested in the community. MVP is committed to helping others and maximizing the use of Microsoft technologies by Microsoft technical community users by speaking, forum Q&A, creating websites, writing blogs, sharing videos, open source projects, organizing conferences, etc.
For more details, please visit the official website:
https://mvp.microsoft.com/en-us

Long press to identify the QR code and follow Microsoft Developer MSDN

Click to learn about UI Automation~