How to investigate objects/types/etc. from Scala REPL?

Question

I've been working with Scala for a while now and have written a 10,000+ line program with it, but I'm still confused by some of the inner workings. I came to Scala from Python after already having intimate familiarity with Java, C and Lisp, but even so it's been slow going, and a huge problem is the frustrating difficulty I've often found when trying to investigate the inner workings of objects/types/classes/etc. using the Scala REPL as compared with Python. In Python you can investigate any object foo (type, object in a global variable, built-in function, etc.) using foo to see what the thing evaluates to, type(foo) to show its type, dir(foo) to tell you the methods you can call on it, and help(foo) to get the built-in documentation. You can even do things like help("re") to find out the documentation on the package named re (which holds regular-expression objects and methods), even though there isn't an object associated with it.

In Scala, you can try and read the documentation online, go look up the source code to the library, etc., but this can often be very difficult for things where you don't know where or even what they are (and it's often a big chunk to bite off, given the voluminous type hierarchy) -- stuff is floating around in various places (package scala, Predef, various implicit conversions, symbols like :: that are nearly impossible to Google). The REPL should be the way to explore directly, but in reality, things are far more mysterious. Say that I've seen a reference to foo somewhere, but I have no idea what it is. There's apparently no such thing as a "guide to systematically investigating Scala thingies with the REPL", but the following is what I've pieced together after a great deal of trial and error:

1. If foo is a value (which presumably includes things stored in variables plus companion objects and other Scala objects), you can evaluate foo directly. This ought to tell you the type and value of the result. Sometimes the result is helpful, sometimes not.
2. If foo is a value, you can use :type foo to get its type. (Not necessarily enlightening.) If you use this on a function call, you get the type of the return value, without calling the function.
3. If foo is a value, you can use foo.getClass to get its class. (Often more enlightening than the previous, but how does an object's class differ from its type?)
4. For a class foo, you can use classOf[foo], although it's not obvious what the result means.
5. Theoretically, you can use :javap foo to disassemble a class -- which should be the most useful of all, but fails entirely and uniformly for me.
6. Sometimes you have to piece things together from error messages.

Example of failure using :javap:

scala> :javap List
Failed: Could not find class bytes for 'List'

Example of enlightening error message:

scala> assert
<console>:8: error: ambiguous reference to overloaded definition,
both method assert in object Predef of type (assertion: Boolean, message: => Any)Unit
and  method assert in object Predef of type (assertion: Boolean)Unit
match expected type ?
              assert
              ^

OK, now let's try a simple example.

scala> 5
res63: Int = 5

scala> :type 5
Int

scala> 5.getClass
res64: java.lang.Class[Int] = int

Simple enough ...

Now, let's try some real cases, where it's not so obvious:

scala> Predef
res65: type = scala.Predef$@3cd41115

scala> :type Predef
type

scala> Predef.getClass
res66: java.lang.Class[_ <: object Predef] = class scala.Predef$

What does this mean? Why is the type of Predef simply type, whereas the class is scala.Predef$? I gather that the $ is the way that companion objects are shoehorned into Java ... but Scala docs on Google tell me that Predef is object Predef extends LowPriorityImplicits -- how can I deduce this from the REPL? And how can I look into what's in it?

OK, let's try another confusing thing:

scala> `::`
res77: collection.immutable.::.type = ::

scala> :type `::`
collection.immutable.::.type

scala> `::`.getClass
res79: java.lang.Class[_ <: object scala.collection.immutable.::] = class scala.collection.immutable.$colon$colon$

scala> classOf[`::`]
<console>:8: error: type :: takes type parameters
              classOf[`::`]
                      ^

scala> classOf[`::`[Int]]
res81: java.lang.Class[::[Int]] = class scala.collection.immutable.$colon$colon

OK, this left me hopelessly confused, and eventually I had to go read the source code to make sense of this all.

So, my questions are:

1. What's the recommended best way from the true Scala experts of using the REPL to make sense of Scala objects, classes, methods, etc., or at least investigate them as best as can be done from the REPL?
2. How do I get :javap working from the REPL for built-in stuff? (Shouldn't it work by default?)

Thanks for any enlightenment.

Answer 1

You mentioned an important point which Scala lacks a bit: the documentation.

The REPL is a fantastic tool, but it is not as fantastic at it can be. There are too much missing features and features which can be improved - some of them are mentioned in your post. Scaladoc is a nice tool, too, but is far away to be perfect. Furthermore lots of code in the API is not yet or too less documented and code examples are often missing. The IDEs are full ob bugs and compared to the possibilities Java IDEs show us they look like some kindergarten toys.

Nevertheless there is a gigantic difference of Scalas current tools compared to the tools available as I started to learn Scala 2-3 years ago. At that time IDEs compiled permanently some trash in the background, the compiler crashed every few minutes and some documentation was absolutely nonexistent. Frequently I got rage attacks and wished death and corruption to Scala authors.

And now? I do not have any of these rage attacks any more. Because the tools we currently have are great although the are not perfect!

There is docs.scala-lang.org, which summarizes a lot of great documentation. There are Tutorials, Cheat-sheets, Glossaries, Guides and a lot of more great stuff. Another great tools is Scalex, which can find even the weirdest operator one can think of. It is Scalas Hoogle and even though it is not yet as good as his great ideal, it is very useful.

Great improvements are coming with Scala2.10 in form of Scalas own Reflection library:

// needs Scala2.10M4
scala> import scala.reflect.runtime.{universe => u}
import scala.reflect.runtime.{universe=>u}

scala> val t = u.typeOf[List[_]]
t: reflect.runtime.universe.Type = List[Any]

scala> t.declarations
res10: Iterable[reflect.runtime.universe.Symbol] = SynchronizedOps(constructor List, method companion, method isEmpty, method head, method tail, method ::, method :::, method reverse_:::, method mapConserve, method ++, method +:, method toList, method take, method drop, method slice, method takeRight, method splitAt, method takeWhile, method dropWhile, method span, method reverse, method stringPrefix, method toStream, method removeDuplicates)

Documentation for the new Reflection library is still missing, but in progress. It allows one to use scalac in an easy way inside of the REPL:

scala> u reify { List(1,2,3) map (_+1) }
res14: reflect.runtime.universe.Expr[List[Int]] = Expr[List[Int]](immutable.this.List.apply(1, 2, 3).map(((x$1) => x$1.$plus(1)))(immutable.this.List.canBuildFrom))

scala> import scala.tools.reflect.ToolBox
import scala.tools.reflect.ToolBox

scala> import scala.reflect.runtime.{currentMirror => m}
import scala.reflect.runtime.{currentMirror=>m}

scala> val tb = m.mkToolBox()
tb: scala.tools.reflect.ToolBox[reflect.runtime.universe.type] = scala.tools.reflect.ToolBoxFactory$ToolBoxImpl@32f7fa37

scala> tb.parseExpr("List(1,2,3) map (_+1)")
res16: tb.u.Tree = List(1, 2, 3).map(((x$1) => x$1.$plus(1)))

scala> tb.runExpr(res16)
res18: Any = List(2, 3, 4)

This is even greater when we want to know how Scala code is translated internally. Formerly wen need to type scala -Xprint:typer -e "List(1,2,3) map (_+1)" to get the internally representation. Furthermore some small improvements found there way to the new release, for example:

scala> :type Predef
scala.Predef.type

Scaladoc will gain some type-hierarchy graph (click on type-hierarchy).

With Macros it is possible now, to improve error messages in a great way. There is a library called expecty, which does this:

// copied from GitHub page
import org.expecty.Expecty

case class Person(name: String = "Fred", age: Int = 42) {
  def say(words: String*) = words.mkString(" ")
}

val person = Person()
val expect = new Expecty()

// Passing expectations

expect {
  person.name == "Fred"
  person.age * 2 == 84
  person.say("Hi", "from", "Expecty!") == "Hi from Expecty!"
}

// Failing expectation

val word1 = "ping"
val word2 = "pong"

expect {
  person.say(word1, word2) == "pong pong"
}

/*
Output:

java.lang.AssertionError:

person.say(word1, word2) == "pong pong"
|      |   |      |      |
|      |   ping   pong   false
|      ping pong
Person(Fred,42)
*/

There is a tool which allows one to find libraries hosted on GitHub, called ls.implicit.ly.

The IDEs now have some semantic highlighting, to show if a member is a object/type/method/whatever. The semantic highlighting feature of ScalaIDE.

The javap feature of the REPL is only a call to the native javap, therefore it is not a very featue-rich tool. You have to fully qualify the name of a module:

scala> :javap scala.collection.immutable.List
Compiled from "List.scala"
public abstract class scala.collection.immutable.List extends scala.collection.AbstractSeq implements scala.collection.immutable.LinearSeq,scala.Product,scala.collection.LinearSeqOptimized{
...

Some time ago I have written a summary of how Scala code is compiled to Bytecode, which offers a lot of things to know.

And the best: This is all done in the last few months!

So, how to use all of these things inside of the REPL? Well, it is not possible ... not yet. ;)

But I can tell you that one day we will have such a REPL. A REPL which shows us documentation if we want to see it. A REPL which let us communicate with it (maybe like lambdabot). A REPL which let us do cool things we still cannot imagine. I don't know when this will be the case, but I know that a lot of stuff was done in the last years and I know even greater stuff will be done in the next years.