Knowledge Acquisition for Goal Prediction in a Multi-User ...

Knowledge Acquisition for Goal Prediction in a Multi-User ...

Knowledge Engineering for Bayesian Networks Ann Nicholson School of Computer Science and Software Engineering Monash University 1 Overview Representing uncertainty Introduction to Bayesian Networks Syntax, semantics, examples The knowledge engineering process Case Studies Seabreeze prediction Intelligent Tutoring Open research questions

2 Sources of Uncertainty Ignorance Inexact observations Non-determinism AI representations Probability theory Dempster-Shafer Fuzzy logic 3 Probability theory for representing uncertainty Assigns a numerical degree of belief between 0 and 1 to facts e.g. it will rain today is T/F. P(it will rain today) = 0.2 prior probability (unconditional)

Posterior probability (conditional) P(it wil rain today | rain is forecast) = 0.8 Bayes Rule: P(H|E) = P(E|H) x P(H) P(E) 4 Bayesian networks Directed acyclic graphs Nodes: random variables, R: it is raining, discrete values T/F T: temperature, cts or discrete variable C: colour, discrete values {red,blue,green} Arcs indicate dependencies (can have causal interpretation) 5 Bayesian networks

Conditional Probability Distribution (CPD) Associated with each variable probability of each state given parent states Jane has the flu FXlu P(Flu=T) = 0.05 TYe P(Te=High|Flu=T) = 0.4 P(Te=High|Flu=F) = 0.01 Models causal relationship Jane has a high temp Models possible sensor error Thermometer temp reading TQh P(Th=High|Te=H) = 0.95 6 P(Th=High|Te=L) = 0.1 BN inference

Evidence: observation of specific state Task: compute the posterior probabilities for query node(s) given evidence. Flu Flu Te Y TYe Th Th Diagnostic inference Causal inference Flu TB Te

Flu Te Th Intercausal inference 7 Mixed inference BN software Commerical packages: Netica, Hugin, Analytica (all with demo versions) Free software: Smile, Genie, JavaBayes, http://HTTP.CS.Berkeley.EDU/~murphyk/Bayes/ bnsoft.html Examples

8 Decision networks Extension to basic BN for decision making Decision nodes Utility nodes EU(Action) = p(o|Action,E) U(o) o choose action with highest expect utility Example 9 Elicitation from experts Variables important variables? values/states?

Structure causal relationships? dependencies/independencies? Parameters (probabilities) quantify relationships and interactions? Preferences (utilities) 10 Expert Elicitation Process These stages are done iteratively Stops when further expert input is no longer cost effective Process is difficult and time consuming. Current BN tools inference engine GUI

BN EXPERT Domain EXPERT Next generation of BN tools? BN TOOLS 11 Knowledge discovery There is much interest in automated methods for learning BNS from data parameters, structure (causal discovery) Computationally complex problem, so current methods have practical limitations e.g. limit number of states, require variable ordering constraints, do not specify all arc directions

Evaluation methods 12 The knowledge engineering process 1. Building the BN variables, structure, parameters, preferences combination of expert elicitation and knowledge discovery 2. Validation/Evaluation case-based, sensitivity analysis, accuracy testing 3. Field Testing alpha/beta testing, acceptance testing 4. Industrial Use collection of statistics 5. Refinement Updating procedures, regression testing 13 Case Study: Intelligent tutoring

Tutoring domain: primary and secondary school students misconceptions about decimals Based on Decimal Comparison Test (DCT) student asked to choose the larger of pairs of decimals different types of pairs reveal different misconceptions ITS System involves computer games involving decimals This research also looks at a combination of expert elicitation and automated methods 14 Expert classification of Decimal Comparison Test (DCT) results expert class ATE AMO MIS AU LWH LZE LRV LU

SDF SRN SU UN 1 0.4 0.35 H H L H L L L L H H H - 2 5.736 5.62 H H L H H

H H H L L L - Item Type 3 4 4.7 0.452 4.08 0.45 H H H L L L L H H H L H H L

H L - 5 0.4 0.3 H H L H H H H L - 6 0.42 0.35 H H L H H L H L 15

The ITS architecture Adaptive Bayesian Network Inputs Student Generic BN model of student Decimal comparison test (optional) Answers Diagnose misconception Predict outcomes Identify most useful information Information about student e.g. age (optional)

Classroom diagnostic test results (optional) Answer Computer Games Hidden number Answer Feedback Answer System Controller Module Sequencing tactics Item Select next item type Decide to present help Decide change to new game

Identify when expertise gained Flying photographer Item type Item Decimaliens New game Help Number between Help . Report on student Classroom Teaching Activities

16 Teacher Expert Elicitation Variables two classification nodes: fine and coarse (mut. ex.) item types: (i) H/M/L (ii) 0-N Structure arcs from classification to item type item types independent given classification Parameters careless mistake (3 different values) expert ignorance: - in table (uniform distribution) 17 Expert Elicited BN 18 Evaluation process

Case-based evaluation experts checked individual cases sometimes, if prior was low, true classification did not have highest posterior (but usually had biggest change in ratio) Adaptiveness evaluation priors changes after each set of evidence Comparison evaluation Differences in classification between BN and expert rule Differences in predictions between different BNs 19 Comparison evaluation Development of measure: same classification, desirable and undesirable re-classification Use item type predictions Investigation of effect of item type granularity

and probability of careless mistake 20 Investigation by Automated methods Classification (using SNOB program, based on MML) Parameters Structure (using CaMML) 21 Results Method Expert Type values 0-N H/M/L 24 DCT 0-N

H/M/L EBN 0-N learned H/M/L CaMML 0-N contr. H/M/L CaMML 0-N uncontr. H/M/L Match 0.22 0.11 0.03 0.22 0.11 0.03 SNOB Avg Avg Avg Avg Avg Avg 77.88 82.93

84.37 80.47 83.91 90.40 79.81 72.06 72.51 95.97 97.63 86.51 83.48 85.15 92.63 Desir. Undes. Change Change 20.39 1.72 15.63 1.44 11.86 3.78 18.71 0.82 13.66 2.42 6.48 3.12

17.60 2.49 16.00 11.94 17.03 10.46 2.36 1.66 1.61 0.75 5.08 8.41 8.12 8.34 5.87 7.92 4.61 2.76 22 Case Study: Seabreeze prediction

2000 Honours project, joint with Bureau of Meteorology (PAKDD2001 paper, TR) BN network built based on existing simple expert rule Several years data available for Sydney seabreezes CaMML and Tetrad-II programs used to learn BNs from data Comparative analysis showed automated methods gave improved predictions. 23 Open Research Questions Tools needed to support expert elicitation Combining expert elicitation and automated methods Evaluation measures and methods Industry adoption of BN technology 24

Recently Viewed Presentations

  • Spinal Analgesia for Palliative Care

    Spinal Analgesia for Palliative Care

    Overview of Spinal Analgesia. Prof Lesley Colvin. Consultant/ Hon Professor in Anaesthesia & Pain Medicine. Cancer pain: is it a problem? ... Epidural/ intrathecal analgesia. Spinous process. Spinal cord. Dorsal root. Ventral root. Vertebral body. Dura mater.
  • Low-carbon Technology in Poland Background and Perspective

    Low-carbon Technology in Poland Background and Perspective

    Use CNG on a large scale as vehicle fuel; Strengthening local markets, particularly, food markets. ... excellent construction material for furniture and other products ... LOW-CARBON TECHNOLOGY IN POLAND BACKGROUND AND PERSPECTIVE Last modified by:
  • Progress Report for the 2010AtSABATH Group Meeting Feng

    Progress Report for the 2010AtSABATH Group Meeting Feng

    Progress Report for the 2010AtSABATH Group Meeting Feng Chen Ann Arbor, June 2004 1. Management 2. Progress and Resources on 5 Genes 3. High-throughput Biochemical Assays Part I: Report on Management NSF Arabidopsis 2010: Functional Analysis of the SABATH Family...
  • Curs PQPI 2007/2008

    Curs PQPI 2007/2008

    Defineixen allò que els estudiants seran capaços de realitzar al finalitzar el curs i els resultats esperats. Un dels elements més importants de la "programació d'una assignatura" Objectives, aims, learning skills, program outcomes, competences, level descriptors, etc... Objectius formatius.
  • Catálogo de Indicação Literária -

    Catálogo de Indicação Literária -

    Pip é nome do personagem central de Grandes Esperanças. Mas Pip também é o nome de um dos personagens principais de O Sr. Pip, livro de autoria do escritor neozelandês Lloyd Jones. A história se passa numa ilha do Atlântico...
  • Multi-year Programme of Work of the Committee COFO

    Multi-year Programme of Work of the Committee COFO

    Two sessions in the MYPOW period: 2016, 2018 with WFW. Meeting calendar coordinated with RFCs and RCs. Standing agenda items defined. Role of the SC in recommending specific items. Points for consideration. The Committee is invited to:
  • Folie 1 -

    Folie 1 -

    Presentazione della ditta McDonalds Slide show di Nicole Lehner & Katrin Binder Azienda filiali in 121 paesi 29,000 ristoranti in tutto il mondo 180 milioni di clienti primo ristorante: Canada ristorante più grande : Beijing (Cina) 20% gestiti direttamente...
  • Macbeth Fair is foul - MR. KEMPNER'S ENGLISH PORTAL (2018-19)

    Macbeth Fair is foul - MR. KEMPNER'S ENGLISH PORTAL (2018-19)

    Good vs. Bad Kingship. ... Overall, are the Witches good or evil? the witches. 3/14/2018 "Fair is foul, and foul is fair:Hover through the fog and filthy air. ... bad versus good kingship, embodied by Macbeth and Duncan, respectively, would...